[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 274843160 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.368455] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.370119] pid_max: default: 32768 minimum: 301 [ 0.371032] Security Framework initialized [ 0.371853] SELinux: Initializing. [ 0.373931] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.376939] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.378856] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.380235] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.381910] Initializing cgroup subsys memory [ 0.382811] Initializing cgroup subsys devices [ 0.383773] Initializing cgroup subsys freezer [ 0.384726] Initializing cgroup subsys net_cls [ 0.385656] Initializing cgroup subsys blkio [ 0.386594] Initializing cgroup subsys perf_event [ 0.387597] Initializing cgroup subsys hugetlb [ 0.388562] Initializing cgroup subsys pids [ 0.389436] Initializing cgroup subsys net_prio [ 0.390442] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.392682] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.393921] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.394962] tlb_flushall_shift: 6 [ 0.395797] FEATURE SPEC_CTRL Present [ 0.396689] FEATURE IBPB_SUPPORT Present [ 0.397487] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.398661] Spectre V2 : Vulnerable [ 0.399327] Speculative Store Bypass: Vulnerable [ 0.400894] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.406696] ACPI: Core revision 20130517 [ 0.408779] ACPI: All ACPI Tables successfully acquired [ 0.409792] ftrace: allocating 30294 entries in 119 pages [ 0.459153] Enabling x2apic [ 0.459803] Enabled x2apic [ 0.460758] Switched APIC routing to physical x2apic. [ 0.463378] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.464772] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.467115] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.468991] ... version: 2 [ 0.469964] ... bit width: 48 [ 0.470807] ... generic registers: 4 [ 0.471607] ... value mask: 0000ffffffffffff [ 0.472698] ... max period: 00007fffffffffff [ 0.473763] ... fixed-purpose events: 3 [ 0.474577] ... event mask: 000000070000000f [ 0.475739] KVM setup paravirtual spinlock [ 0.478509] smpboot: Booting Node 0, Processors #1[ 0.479821] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.482961] KVM setup async PF for cpu 1 [ 0.483765] kvm-stealtime: cpu 1, msr 13e2935c0 #2[ 0.485687] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.488795] KVM setup async PF for cpu 2 [ 0.489195] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock #3 OK [ 0.490853] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.491887] Brought up 4 CPUs [ 0.491933] KVM setup async PF for cpu 3 [ 0.491939] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.495557] smpboot: Max logical packages: 1 [ 0.496443] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.500007] devtmpfs: initialized [ 0.500862] x86/mm: Memory block size: 128MB [ 0.504624] EVM: security.selinux [ 0.505348] EVM: security.ima [ 0.505966] EVM: security.capability [ 0.508170] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.509606] NET: Registered protocol family 16 [ 0.510669] cpuidle: using governor haltpoll [ 0.511733] ACPI: bus type PCI registered [ 0.512497] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.513830] PCI: Using configuration type 1 for base access [ 0.514902] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.521479] ACPI: Added _OSI(Module Device) [ 0.522398] ACPI: Added _OSI(Processor Device) [ 0.523255] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.524241] ACPI: Added _OSI(Processor Aggregator Device) [ 0.525269] ACPI: Added _OSI(Linux-Dell-Video) [ 0.528903] ACPI: Interpreter enabled [ 0.529650] ACPI: (supports S0 S3 S4 S5) [ 0.530405] ACPI: Using IOAPIC for interrupt routing [ 0.531379] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.533119] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.538047] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.539223] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.540466] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.541764] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.544817] acpiphp: Slot [2] registered [ 0.545862] acpiphp: Slot [3] registered [ 0.546800] acpiphp: Slot [4] registered [ 0.548070] acpiphp: Slot [5] registered [ 0.549077] acpiphp: Slot [6] registered [ 0.550101] acpiphp: Slot [7] registered [ 0.551133] acpiphp: Slot [8] registered [ 0.552127] acpiphp: Slot [9] registered [ 0.553287] acpiphp: Slot [10] registered [ 0.554352] acpiphp: Slot [11] registered [ 0.555413] acpiphp: Slot [12] registered [ 0.556384] acpiphp: Slot [13] registered [ 0.557340] acpiphp: Slot [14] registered [ 0.558323] acpiphp: Slot [15] registered [ 0.559273] acpiphp: Slot [16] registered [ 0.560472] acpiphp: Slot [17] registered [ 0.561365] acpiphp: Slot [18] registered [ 0.562529] acpiphp: Slot [19] registered [ 0.563614] acpiphp: Slot [20] registered [ 0.564641] acpiphp: Slot [21] registered [ 0.565649] acpiphp: Slot [22] registered [ 0.566544] acpiphp: Slot [23] registered [ 0.567577] acpiphp: Slot [24] registered [ 0.568447] acpiphp: Slot [25] registered [ 0.569502] acpiphp: Slot [26] registered [ 0.570377] acpiphp: Slot [27] registered [ 0.571390] acpiphp: Slot [28] registered [ 0.572277] acpiphp: Slot [29] registered [ 0.573303] acpiphp: Slot [30] registered [ 0.574162] acpiphp: Slot [31] registered [ 0.575146] PCI host bridge to bus 0000:00 [ 0.576027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.577498] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.579205] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.580609] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.582047] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.584647] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.599526] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.602247] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.603644] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.605060] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.607050] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.608452] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.769666] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.771201] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.772895] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.774321] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.776605] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.779790] vgaarb: loaded [ 0.780589] SCSI subsystem initialized [ 0.781489] ACPI: bus type USB registered [ 0.782482] usbcore: registered new interface driver usbfs [ 0.783582] usbcore: registered new interface driver hub [ 0.784883] usbcore: registered new device driver usb [ 0.786196] PCI: Using ACPI for IRQ routing [ 0.787573] NetLabel: Initializing [ 0.788302] NetLabel: domain hash size = 128 [ 0.789209] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.790211] NetLabel: unlabeled traffic allowed by default [ 0.791645] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.792744] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.797196] amd_nb: Cannot enumerate AMD northbridges [ 0.798439] Switched to clocksource kvm-clock [ 0.813088] pnp: PnP ACPI init [ 0.813812] ACPI: bus type PNP registered [ 0.816099] pnp: PnP ACPI: found 6 devices [ 0.817288] ACPI: bus type PNP unregistered [ 0.827682] NET: Registered protocol family 2 [ 0.829859] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 0.832163] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 0.834888] TCP: Hash tables configured (established 32768 bind 32768) [ 0.836611] TCP: reno registered [ 0.838023] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 0.840546] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 0.843367] NET: Registered protocol family 1 [ 0.844925] RPC: Registered named UNIX socket transport module. [ 0.846348] RPC: Registered udp transport module. [ 0.847664] RPC: Registered tcp transport module. [ 0.848620] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.850119] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.851685] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.853112] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.854662] Unpacking initramfs... [ 2.257086] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 2.260115] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.262655] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 2.266142] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 2.269284] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 2.271454] RAPL PMU: hw unit of domain package 2^-0 Joules [ 2.273785] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 2.277404] cryptomgr_test (50) used greatest stack depth: 14480 bytes left [ 2.277839] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 2.277886] Initialise system trusted keyring [ 2.314092] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 2.316483] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 2.323120] zpool: loaded [ 2.324219] zbud: loaded [ 2.325738] VFS: Disk quotas dquot_6.6.0 [ 2.327220] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.329905] NFS: Registering the id_resolver key type [ 2.331468] Key type id_resolver registered [ 2.332526] Key type id_legacy registered [ 2.333558] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 2.335363] Key type big_key registered [ 2.338233] cryptomgr_test (56) used greatest stack depth: 13968 bytes left [ 2.342505] NET: Registered protocol family 38 [ 2.344496] Key type asymmetric registered [ 2.346877] Asymmetric key parser 'x509' registered [ 2.349230] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 2.352067] io scheduler noop registered [ 2.353608] io scheduler deadline registered (default) [ 2.355579] io scheduler cfq registered [ 2.356916] io scheduler mq-deadline registered [ 2.358045] io scheduler kyber registered [ 2.360912] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 2.362332] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 2.364130] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.366013] ACPI: Power Button [PWRF] [ 2.367182] GHES: HEST is not enabled! [ 2.434309] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 2.499847] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 2.617154] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 2.673377] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 2.785213] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.813412] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 2.842273] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 2.845969] Non-volatile memory driver v1.3 [ 2.847633] Linux agpgart interface v0.103 [ 2.849501] crash memory driver: version 1.1 [ 2.851474] nbd: registered device at major 43 [ 2.862768] virtio_blk virtio1: [vda] 57680 512-byte logical blocks (29.5 MB/28.1 MiB) [ 2.875721] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.886563] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.897542] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.909436] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.922541] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.928245] rdac: device handler registered [ 2.930000] hp_sw: device handler registered [ 2.931385] emc: device handler registered [ 2.932368] libphy: Fixed MDIO Bus: probed [ 2.935877] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 2.937996] ehci-pci: EHCI PCI platform driver [ 2.939549] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 2.941403] ohci-pci: OHCI PCI platform driver [ 2.942447] uhci_hcd: USB Universal Host Controller Interface driver [ 2.943856] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.946682] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.948221] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.949675] mousedev: PS/2 mouse device common for all mice [ 2.952032] rtc_cmos 00:05: RTC can wake from S4 [ 2.952201] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.959707] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 2.961957] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.965710] hidraw: raw HID events driver (C) Jiri Kosina [ 2.967233] usbcore: registered new interface driver usbhid [ 2.968704] usbhid: USB HID core driver [ 2.969917] drop_monitor: Initializing network drop monitor service [ 2.971814] Netfilter messages via NETLINK v0.30. [ 2.973028] TCP: cubic registered [ 2.974326] Initializing XFRM netlink socket [ 2.976469] NET: Registered protocol family 10 [ 2.979130] NET: Registered protocol family 17 [ 2.980739] Key type dns_resolver registered [ 2.982916] mce: Using 10 MCE banks [ 2.984734] Loading compiled-in X.509 certificates [ 2.987806] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 2.990952] registered taskstats version 1 [ 2.994401] modprobe (71) used greatest stack depth: 13456 bytes left [ 2.999531] Key type trusted registered [ 3.005459] Key type encrypted registered [ 3.006668] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 3.009730] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 3.012337] rtc_cmos 00:05: setting system clock to 2024-06-05 03:03:13 UTC (1717556593) [ 3.015075] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 3.016978] Write protecting the kernel read-only data: 12288k [ 3.018546] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 3.020307] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 3.028878] random: systemd: uninitialized urandom read (16 bytes read) [ 3.032667] random: systemd: uninitialized urandom read (16 bytes read) [ 3.035015] random: systemd: uninitialized urandom read (16 bytes read) [ 3.039315] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 3.045635] systemd[1]: Detected virtualization kvm. [ 3.047258] systemd[1]: Detected architecture x86-64. [ 3.048977] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-572.el7 (Initramfs)! [ 3.054595] systemd[1]: No hostname configured. [ 3.056304] systemd[1]: Set hostname to . [ 3.058379] random: systemd: uninitialized urandom read (16 bytes read) [ 3.061033] systemd[1]: Initializing machine ID from random generator. [ 3.108755] dracut-rootfs-g (85) used greatest stack depth: 13264 bytes left [ 3.111796] random: systemd: uninitialized urandom read (16 bytes read) [ 3.113980] random: systemd: uninitialized urandom read (16 bytes read) [ 3.116298] random: systemd: uninitialized urandom read (16 bytes read) [ 3.118801] random: systemd: uninitialized urandom read (16 bytes read) [ 3.122519] random: systemd: uninitialized urandom read (16 bytes read) [ 3.124385] random: systemd: uninitialized urandom read (16 bytes read) [ 3.133708] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.136632] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.140325] systemd[1]: Created slice Root Slice. [ OK ] Created slice Root Slice. [ 3.143197] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.147086] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.150829] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.154776] systemd[1]: Reached target Sockets. [ OK ] Reached target Sockets. [ 3.158318] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.163778] systemd[1]: Created slice System Slice. [ OK ] Created slice System Slice. [ 3.169084] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 3.173919] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ 3.181179] systemd[1]: Starting Load Kernel Modules... Starting Load Kernel Modules... [ 3.185630] systemd[1]: Starting Journal Service... Starting Journal Service... [ 3.188139] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.192419] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ 3.196963] systemd[1]: Started Setup Virtual Console. [ OK ] Started Setup Virtual Console. [ 3.202950] systemd[1]: Started Create list of required static device nodes for the current kernel. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ 3.209163] systemd[1]: Started Load Kernel Modules. [ OK ] Started Load Kernel Modules. [ 3.215385] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ 3.220754] systemd[1]: Starting Create Static Device Nodes in /dev... Starting Create Static Device Nodes in /dev... [ 3.228223] systemd[1]: Started Journal Service. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ 3.277535] tsc: Refined TSC clocksource calibration: 2399.980 MHz [ 3.360832] random: fast init done [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Configuration File System... [ OK ] Mounted Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ 3.693283] scsi host0: ata_piix Starting Show Plymouth Boot Screen... [ 3.700719] scsi host1: ata_piix [ 3.702647] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 3.704657] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 Starting dracut initqueue hook... [ OK ] Started Show Plymouth Boot Screen. [ OK ] Reached target Paths. [ OK ] Started Forward Password Requests to Plymouth Directory Watch. [ OK ] Reached target Basic System. %G[ 3.794169] ip (320) used greatest stack depth: 13080 bytes left [ 3.795258] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 3.845077] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 3.847917] ip (343) used greatest stack depth: 12464 bytes left [ 3.937515] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 6.135305] dracut-initqueue[287]: RTNETLINK answers: File exists [ 6.323632] dracut-initqueue[287]: bs=4096, sz=32212254720 bytes [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ 6.850764] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Starting Plymouth switch root service... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Swap. [ OK ] Stopped target Paths. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Load Kernel Modules. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Plymouth switch root service. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Reached target Switch Root. Starting Switch Root... [ 7.210415] systemd-journald[105]: Received SIGTERM from PID 1 (systemd). [ 7.396069] SELinux: Disabled at runtime. [ 7.451133] ip_tables: (C) 2000-2006 Netfilter Core Team [ 7.453386] systemd[1]: Inserted module 'ip_tables' Welcome to CentOS Linux 7 (Core)! [ OK ] Stopped Switch Root. [ OK ] Stopped Journal Service. Starting Journal Service... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. Starting Create list of required st... nodes for the current kernel... [ OK ] Listening on /dev/initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. Starting Read and set NIS domainname from /etc/sysconfig/network... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. Mounting Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-selinux\x2dpol...grate\x2dlocal\x2dchanges.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Delayed Shutdown Socket. Starting Load Kernel Modules... Starting Remount Root and Kernel File Systems... Starting Set Up Additional Binary Formats... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Debug File System. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Load Kernel Modules. Starting Apply Kernel Variables... Starting Create Static Device Nodes in /dev... [ OK ] Started Read and set NIS domainname from /etc/sysconfig/network. Mounting Arbitrary Executable File Formats File System... [ OK ] Started Journal Service. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Arbitrary Executable File Formats File System. [ 7.739809] mount.nfs (585) used greatest stack depth: 12448 bytes left [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Set Up Additional Binary Formats. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 7.772325] systemd-journald[565]: Received request to flush runtime journal from PID 1 [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Kernel Device Manager. [ 7.865105] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ OK ] Found device /dev/ttyS1. [ 7.892221] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ OK ] Found device /dev/ttyS0. [ 7.901149] cryptd: max_cpu_qlen set to 1000 [ OK ] Found device /dev/vda. Mounting /home/green/git/lustre-release... [ OK [ 7.937187] AVX version of gcm_enc/dec engaged. [ 7.939204] AES CTR mode by8 optimization enabled ] Found device /dev/disk/by-label/SWAP. Activating swap /dev/disk/by-label/SWAP... [ 7.972910] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap.[ 7.981777] squashfs: version 4.0 (2009/01/31) Phillip Lougher [[ 7.991657] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni)  OK ] Mounted /home/g[ 7.994923] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) reen/git/lustre-release. %G[ 8.088608] EDAC MC: Ver: 3.0.0 [ 8.093080] EDAC sbridge: Ver: 1.1.2 [ 9.243352] mount.nfs (771) used greatest stack depth: 10704 bytes left [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Tell Plymouth To Write Out Runtime Data... Starting Preprocess NFS configuration... Starting Rebuild Journal Catalog... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ OK ] Started Preprocess NFS configuration. [FAILED] Failed to start Rebuild Journal Catalog. See 'systemctl status systemd-journal-catalog-update.service' for details. [ OK ] Started Mark the need to relabel after reboot. [FAILED] Failed to start Create Volatile Files and Directories. See 'systemctl status systemd-tmpfiles-setup.service' for details. Starting Update UTMP about System Boot/Shutdown... Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Flexible branding. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Dump dmesg to /var/log/dmesg... Starting GSSAPI Proxy Daemon... Starting Network Manager... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Dump dmesg to /var/log/dmesg. [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting /etc/rc.d/rc.local Compatibility... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started /etc/rc.d/rc.local Compatibility. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... Starting Wait for Plymouth Boot Screen to Quit... Starting Terminate Plymouth Boot Screen... [ OK ] Started Network Manager Script Dispatcher Service. CentOS Linux 7 (Core) Kernel 3.10.0-7.9-debug on an x86_64 oleg345-server login: [ 20.908761] libcfs: loading out-of-tree module taints kernel. [ 20.910988] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 20.935034] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 25.533732] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local[ 25.719409] alg: No test for adler32 (adler32-zlib) [ 26.470561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 28.237722] Key type lgssc registered [ 28.902471] icp: module license 'CDDL' taints kernel. [ 28.904076] Disabling lock debugging due to kernel taint [ 31.369527] ZFS: Loaded module v0.8.6-1, ZFS pool version 5000, ZFS filesystem version 5 [ 33.285070] vdc: vdc1 vdc9 [ 35.551650] vde: vde1 vde9 [ 37.909282] vdf: vdf1 vdf9 [ 41.537894] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 43.280061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'Lustre: lustre-MDT0000: new disk, initializing[ 44.504606] random: crng init done [ 44.622285] mount.lustre (6915) used greatest stack depth: 10184 bytes left [ 45.373833] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck allLustre: lustre-OST0000: new disk, initializing [ 50.057274] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck allLustre: lustre-OST0001: new disk, initializing [ 54.884274] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck allLustre: DEBUG MARKER: Using TIMEOUT=20 [ 70.940341] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing check_logdir /tmp/testlogs/Lustre: DEBUG MARKER: oleg345-server.virtnet: executing yml_node [ 72.650757] Lustre: DEBUG MARKER: Client: 2.15.63.95Lustre: DEBUG MARKER: MDS: 2.15.63.95 [ 73.827800] Lustre: DEBUG MARKER: OSS: 2.15.63.95Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Tue Jun 4 23:04:23 EDT 2024 [ 77.970721] Lustre: DEBUG MARKER: excepting tests: 110f 131b 59 36Lustre: DEBUG MARKER: oleg345-client.virtnet: executing check_config_client /mnt/lustre [ 84.088761] Lustre: DEBUG MARKER: Using TIMEOUT=20Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 86.230641] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 23:04:35 (1717556675)LustreError: 15168:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 87.271169] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000Lustre: Failing over lustre-MDT0000 [ 87.903142] Lustre: server umount lustre-MDT0000 completeLustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 99.951449] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to completeLustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 104.955512] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 104.976517] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556679/real 1717556679] req@ffff880134e75880 x1800988647244928/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556695 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 108.859740] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556679/real 1717556679] req@ffff880134e76300 x1800988647244864/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556695 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 109.559231] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 109.895512] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 109.969560] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556684/real 1717556684] req@ffff880134e77800 x1800988647245120/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556700 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 110.013766] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556684/real 1717556684] req@ffff880134e74e00 x1800988647245056/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556700 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 112.663628] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 113.213113] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 23:05:02 (1717556702) [ 113.229174] Lustre: Failing over lustre-OST0000 [ 113.897742] Lustre: server umount lustre-OST0000 complete [ 113.898997] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 113.901040] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 114.980582] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 114.985846] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556689/real 1717556689] req@ffff8800a801e300 x1800988647245312/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556705 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 117.880232] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 119.993792] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 122.884334] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 124.845192] mount.lustre (17821) used greatest stack depth: 10176 bytes left [ 126.027640] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 126.598465] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 128.256589] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 128.613043] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 131.441585] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 132.174668] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 23:05:20 (1717556720) [ 132.402942] LustreError: 18997:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 132.935851] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 133.047292] Lustre: Failing over lustre-MDT0000 [ 144.773767] Lustre: server umount lustre-MDT0000 complete [ 144.873804] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 144.873889] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 145.010381] mount.lustre (19879) used greatest stack depth: 10168 bytes left [ 145.772587] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 146.319549] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 146.322738] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 150.949579] Lustre: lustre-MDT0000: Denying connection for new client e8fa1c2a-aa1e-4252-b3fb-cc7b8fbb0a64 (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 150.977504] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556725/real 1717556725] req@ffff880131d06d80 x1800988647255168/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556741 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 151.332049] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556725/real 1717556725] req@ffff880131d07b80 x1800988647255232/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556741 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 155.985620] Lustre: lustre-MDT0000: Denying connection for new client e8fa1c2a-aa1e-4252-b3fb-cc7b8fbb0a64 (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 155.985697] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556730/real 1717556730] req@ffff880131d05500 x1800988647255424/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556746 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 156.339578] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556730/real 1717556730] req@ffff880131d04a80 x1800988647255360/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556746 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 161.348294] Lustre: lustre-MDT0000: Denying connection for new client e8fa1c2a-aa1e-4252-b3fb-cc7b8fbb0a64 (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 166.355510] Lustre: lustre-MDT0000: Denying connection for new client e8fa1c2a-aa1e-4252-b3fb-cc7b8fbb0a64 (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:44 [ 171.363495] Lustre: lustre-MDT0000: Denying connection for new client e8fa1c2a-aa1e-4252-b3fb-cc7b8fbb0a64 (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:39 [ 176.371530] Lustre: lustre-MDT0000: Denying connection for new client e8fa1c2a-aa1e-4252-b3fb-cc7b8fbb0a64 (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:34 [ 181.379544] Lustre: lustre-MDT0000: Denying connection for new client e8fa1c2a-aa1e-4252-b3fb-cc7b8fbb0a64 (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:29 [ 186.387556] Lustre: lustre-MDT0000: Denying connection for new client e8fa1c2a-aa1e-4252-b3fb-cc7b8fbb0a64 (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:24 [ 191.395567] Lustre: lustre-MDT0000: Denying connection for new client e8fa1c2a-aa1e-4252-b3fb-cc7b8fbb0a64 (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:19 [ 196.403540] Lustre: lustre-MDT0000: Denying connection for new client e8fa1c2a-aa1e-4252-b3fb-cc7b8fbb0a64 (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:14 [ 201.411623] Lustre: lustre-MDT0000: Denying connection for new client e8fa1c2a-aa1e-4252-b3fb-cc7b8fbb0a64 (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:09 [ 205.426543] Lustre: lustre-MDT0000: Denying connection for new client e8fa1c2a-aa1e-4252-b3fb-cc7b8fbb0a64 (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:04 [ 205.431958] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 205.433465] Lustre: 20045:0:(/home/green/git/lustre-release/lustre/obdclass/genops.c:1533:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 6f5fc865-596a-4812-9ca9-898b5ab877e2@ [ 209.009046] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 209.720659] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 23:06:38 (1717556798) [ 209.947783] LustreError: 21086:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 210.467836] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 210.584261] Lustre: Failing over lustre-MDT0000 [ 222.158508] Lustre: server umount lustre-MDT0000 complete [ 222.260791] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 222.267480] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 222.304320] mount.lustre (21982) used greatest stack depth: 10104 bytes left [ 223.017159] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 223.557373] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.560774] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 228.563530] Lustre: lustre-MDT0000: Denying connection for new client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 231.089578] Lustre: lustre-MDT0000: Denying connection for new client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 231.289577] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556805/real 1717556805] req@ffff880093e1a680 x1800988647266304/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556821 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 233.571473] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556805/real 1717556805] req@ffff880093e1aa00 x1800988647266368/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556821 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 236.093578] Lustre: lustre-MDT0000: Denying connection for new client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 236.297567] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556810/real 1717556810] req@ffff880093e1b480 x1800988647266560/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556826 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 238.579699] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556810/real 1717556810] req@ffff880093e1b100 x1800988647266496/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556826 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 243.587656] Lustre: lustre-MDT0000: Denying connection for new client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:44 [ 248.595545] Lustre: lustre-MDT0000: Denying connection for new client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:39 [ 253.603325] Lustre: lustre-MDT0000: Denying connection for new client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:34 [ 258.611672] Lustre: lustre-MDT0000: Denying connection for new client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:29 [ 263.619550] Lustre: lustre-MDT0000: Denying connection for new client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:24 [ 268.627612] Lustre: lustre-MDT0000: Denying connection for new client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:19 [ 273.636118] Lustre: lustre-MDT0000: Denying connection for new client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:14 [ 278.643412] Lustre: lustre-MDT0000: Denying connection for new client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:09 [ 283.426515] Lustre: lustre-MDT0000: Denying connection for new client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:04 [ 283.432624] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 283.434593] Lustre: 22081:0:(/home/green/git/lustre-release/lustre/obdclass/genops.c:1533:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client e8fa1c2a-aa1e-4252-b3fb-cc7b8fbb0a64@ [ 286.292485] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 287.030771] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 23:07:55 (1717556875) [ 287.262236] LustreError: 23147:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 287.756759] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 287.862571] Lustre: Failing over lustre-MDT0000 [ 299.438911] Lustre: server umount lustre-MDT0000 complete [ 299.532964] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 299.534529] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.289728] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 302.006993] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 302.542803] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 302.905987] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 305.496301] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 306.223424] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 23:08:14 (1717556894) [ 306.447589] LustreError: 25197:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 306.931666] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 307.027790] Lustre: Failing over lustre-MDT0000 [ 307.553485] Lustre: server umount lustre-MDT0000 complete [ 307.555064] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556882/real 1717556882] req@ffff88008f241f80 x1800988647276096/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556898 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 312.532609] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556882/real 1717556882] req@ffff88008f242a00 x1800988647276160/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556898 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 312.563018] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556887/real 1717556887] req@ffff88008f242300 x1800988647276288/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556903 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 318.795158] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556887/real 1717556887] req@ffff88008f240000 x1800988647276352/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556903 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 318.895298] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 318.899138] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 319.761337] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.500206] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 322.067684] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 322.470183] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 325.085861] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 325.841556] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 23:08:34 (1717556914) [ 325.905552] LustreError: 27416:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 325.913534] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556900/real 1717556900] req@ffff88009d7a3100 x1800988647281216/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556916 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 326.064898] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556900/real 1717556900] req@ffff88009d7a0380 x1800988647281280/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556916 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 326.545728] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 326.639841] Lustre: Failing over lustre-MDT0000 [ 330.545576] Lustre: server umount lustre-MDT0000 complete [ 330.921539] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556905/real 1717556905] req@ffff88009d7a2300 x1800988647281408/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556921 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 338.234248] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556905/real 1717556905] req@ffff88009d7a3b80 x1800988647281472/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556921 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 338.347598] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.353621] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.393222] mount.lustre (28285) used greatest stack depth: 9792 bytes left [ 338.756458] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.100988] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 341.301835] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 341.660194] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 344.227727] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 344.968163] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 23:08:53 (1717556933) [ 345.196453] LustreError: 29521:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 345.353510] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 345.356480] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556919/real 1717556919] req@ffff88012ec98a80 x1800988647286464/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556935 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 345.695884] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556919/real 1717556919] req@ffff88012ec98e00 x1800988647286528/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556935 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 345.800243] Lustre: Failing over lustre-MDT0000 [ 349.694569] Lustre: server umount lustre-MDT0000 complete [ 350.377606] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556924/real 1717556924] req@ffff88012ec9a300 x1800988647286656/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556940 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 357.335836] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556924/real 1717556924] req@ffff88012ec9a680 x1800988647286720/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556940 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 357.435929] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 357.435978] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 358.204672] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 358.788400] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 360.411184] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 360.760107] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 363.280003] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 363.980629] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 23:09:12 (1717556952) [ 364.202194] LustreError: 31648:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 364.385507] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 364.473494] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556938/real 1717556938] req@ffff880131d06300 x1800988647291840/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556954 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 364.693664] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556938/real 1717556938] req@ffff880131d07100 x1800988647291904/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556954 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 364.797569] Lustre: Failing over lustre-MDT0000 [ 369.481558] Lustre: server umount lustre-MDT0000 complete [ 369.481566] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556943/real 1717556943] req@ffff880131d05f80 x1800988647292096/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556959 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 376.412448] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556943/real 1717556943] req@ffff880131d04a80 x1800988647292032/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556959 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 376.508544] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 376.508558] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 377.253738] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 378.820262] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 379.538989] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 379.943462] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 382.513540] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 382.529515] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556957/real 1717556957] req@ffff8800a6c17800 x1800988647297088/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556973 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 382.548776] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556957/real 1717556957] req@ffff8800a6c16d80 x1800988647297024/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556973 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 382.797205] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 23:09:31 (1717556971) [ 384.540319] LustreError: 32506:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff880094bc2d80 x1800988640982656/t38654705666(0) o35->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:695/0 lens 392/456 e 0 to 0 dl 1717556990 ref 1 fl Interpret:/200/0 rc 0/0 job:'openfile.0' uid:0 gid:0 [ 384.781406] LustreError: 1419:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 385.292519] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 385.403130] Lustre: Failing over lustre-MDT0000 [ 387.537558] Lustre: server umount lustre-MDT0000 complete [ 387.537565] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556962/real 1717556962] req@ffff8800a6c16680 x1800988647297280/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556978 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 396.986826] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556962/real 1717556962] req@ffff8800a6c15c00 x1800988647297216/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556978 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 397.852389] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 398.870167] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 398.885838] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 400.056690] Lustre: 2260:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800a7a2d180 x1800988640982656/t38654705666(0) o35->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:711/0 lens 392/456 e 0 to 0 dl 1717557006 ref 1 fl Interpret:/202/0 rc 0/0 job:'openfile.0' uid:0 gid:0 [ 400.418767] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 402.105816] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 402.105818] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 402.999073] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 403.105541] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 23:09:52 (1717556992) [ 403.105546] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556977/real 1717556977] req@ffff8800a7a2d880 x1800988647302144/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556993 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 403.727608] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556977/real 1717556977] req@ffff8800a7a2dc00 x1800988647302208/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556993 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 403.954953] LustreError: 3502:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 404.441905] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 404.558252] Lustre: Failing over lustre-MDT0000 [ 408.113544] Lustre: server umount lustre-MDT0000 complete [ 408.113572] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556982/real 1717556982] req@ffff8800a7a2ed80 x1800988647302400/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556998 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 413.121567] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556982/real 1717556982] req@ffff8800a7a2ea00 x1800988647302336/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556998 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 413.123505] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556987/real 1717556987] req@ffff8800a7a2fb80 x1800988647302528/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557003 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 416.147715] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556987/real 1717556987] req@ffff8800a7a2c000 x1800988647302592/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557003 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 416.258835] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 416.262397] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 417.031823] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 418.757269] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 419.303688] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 419.665964] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 422.223332] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 422.930782] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 23:10:11 (1717557011) [ 423.151376] LustreError: 5623:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 423.263526] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 423.289480] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556997/real 1717556997] req@ffff88009d4a0380 x1800988647307392/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557013 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 424.085792] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556997/real 1717556997] req@ffff88009d4a1880 x1800988647307328/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557013 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 424.200615] Lustre: Failing over lustre-MDT0000 [ 428.085590] Lustre: server umount lustre-MDT0000 complete [ 428.297669] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557002/real 1717557002] req@ffff880094bc2d80 x1800988647307584/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557018 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 435.778487] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557002/real 1717557002] req@ffff88009d4a2a00 x1800988647307520/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557018 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 435.893436] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 435.897093] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 436.653499] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 438.378677] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 438.919004] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 439.280819] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 441.851366] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 441.921512] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 23:10:31 (1717557031) [ 442.305559] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557016/real 1717557016] req@ffff88009a274a80 x1800988647312384/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557032 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 442.583271] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557016/real 1717557016] req@ffff88009a275180 x1800988647312448/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557032 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 442.815837] LustreError: 7849:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 443.775835] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 443.883557] Lustre: Failing over lustre-MDT0000 [ 446.929531] Lustre: server umount lustre-MDT0000 complete [ 446.929536] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557021/real 1717557021] req@ffff88009a275880 x1800988647312576/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557037 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 455.446342] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557021/real 1717557021] req@ffff88009a275c00 x1800988647312640/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557037 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 455.557562] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 455.557591] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 456.324204] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 458.054259] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 458.603205] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 458.964703] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 461.577619] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 461.584490] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 23:10:51 (1717557051) [ 461.585520] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557036/real 1717557036] req@ffff8800a8160700 x1800988647317504/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557052 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 462.302514] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557036/real 1717557036] req@ffff8800a8160e00 x1800988647317440/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557052 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 462.536002] LustreError: 10154:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 463.084217] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 463.202639] Lustre: Failing over lustre-MDT0000 [ 466.593572] Lustre: server umount lustre-MDT0000 complete [ 467.083543] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557041/real 1717557041] req@ffff8800a8161f80 x1800988647317696/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557057 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 474.821281] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557041/real 1717557041] req@ffff8800a8161c00 x1800988647317632/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557057 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 474.921248] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 474.921257] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.700149] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 477.425140] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 478.081134] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 478.439661] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 481.088008] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 481.860650] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 23:11:10 (1717557070) [ 481.945524] LustreError: 12326:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 481.945545] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557056/real 1717557056] req@ffff88012d4aaa00 x1800988647322560/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557072 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 482.092499] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557056/real 1717557056] req@ffff88012d4a8e00 x1800988647322624/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557072 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 482.613875] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 482.722845] Lustre: Failing over lustre-MDT0000 [ 486.622556] Lustre: server umount lustre-MDT0000 complete [ 486.953560] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557061/real 1717557061] req@ffff88012d4a9c00 x1800988647322816/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557077 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 494.358973] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557061/real 1717557061] req@ffff88012d4aad80 x1800988647322752/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557077 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 494.454622] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 494.454650] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.197667] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.896609] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 497.419661] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 497.756640] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 500.323214] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 500.923501] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 23:11:29 (1717557089) [ 501.033533] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557075/real 1717557075] req@ffff8800a13bd180 x1800988647328000/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557091 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 501.261692] LustreError: 14405:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 502.032485] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 502.538733] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557075/real 1717557075] req@ffff8800a13bce00 x1800988647327936/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557091 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 502.648845] Lustre: Failing over lustre-MDT0000 [ 505.538593] Lustre: server umount lustre-MDT0000 complete [ 506.489629] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557080/real 1717557080] req@ffff8800a13bdf80 x1800988647328128/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557096 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 514.216612] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557080/real 1717557080] req@ffff8800a13bed80 x1800988647328192/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557096 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 514.308073] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 514.309227] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 515.058276] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.795527] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 519.699656] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 520.078593] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 520.337510] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 520.337533] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557094/real 1717557094] req@ffff880131d5aa00 x1800988647334592/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557110 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 525.345560] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557094/real 1717557094] req@ffff880131d5b100 x1800988647334656/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557110 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 525.345577] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557099/real 1717557099] req@ffff880131d59880 x1800988647334848/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557115 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 527.556592] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557099/real 1717557099] req@ffff880131d59180 x1800988647334784/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557115 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 528.311266] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 23:11:56 (1717557116) [ 528.557970] LustreError: 16706:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 529.041791] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 529.132999] Lustre: Failing over lustre-MDT0000 [ 540.691183] Lustre: server umount lustre-MDT0000 complete [ 540.780105] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 540.780119] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 541.540521] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.794582] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 544.795509] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557119/real 1717557119] req@ffff88012f39f480 x1800988647368640/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557135 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 549.097379] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557119/real 1717557119] req@ffff88012f39d500 x1800988647368704/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557135 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 549.670300] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 549.801589] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 550.041684] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557124/real 1717557124] req@ffff88012f39c000 x1800988647368896/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557140 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 550.141563] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 554.680494] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557124/real 1717557124] req@ffff88012f39c380 x1800988647368832/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557140 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 555.153524] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 23:12:24 (1717557144) [ 555.153531] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557129/real 1717557129] req@ffff88012f39c700 x1800988647369024/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557145 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 555.432464] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557129/real 1717557129] req@ffff88012f39d880 x1800988647369088/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557145 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 555.662414] LustreError: 18993:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 556.148751] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 556.245977] Lustre: Failing over lustre-MDT0000 [ 567.791422] Lustre: server umount lustre-MDT0000 complete [ 567.903730] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 567.903731] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 568.644478] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 569.140164] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 570.849045] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 571.198799] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 573.791539] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 574.539370] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 23:12:43 (1717557163) [ 574.764761] LustreError: 21114:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 575.243804] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 575.356760] Lustre: Failing over lustre-MDT0000 [ 576.937604] Lustre: server umount lustre-MDT0000 complete [ 577.243653] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557151/real 1717557151] req@ffff8800953b8380 x1800988647388864/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557167 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 581.945576] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557151/real 1717557151] req@ffff8800953bbb80 x1800988647388928/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557167 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 581.945580] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557156/real 1717557156] req@ffff8800953bb100 x1800988647389120/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557172 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 586.936183] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557156/real 1717557156] req@ffff8800953b9180 x1800988647389056/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557172 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 587.057211] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 587.057504] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 587.820646] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.172358] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 590.001181] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 590.360285] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 592.944823] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 593.682127] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 23:13:02 (1717557182) [ 593.902742] LustreError: 23249:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 594.066546] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 594.073483] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557168/real 1717557168] req@ffff8800937db100 x1800988647393984/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557184 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 594.420630] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557168/real 1717557168] req@ffff8800937dad80 x1800988647393920/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557184 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 594.522180] Lustre: Failing over lustre-MDT0000 [ 599.081540] Lustre: server umount lustre-MDT0000 complete [ 599.420554] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557173/real 1717557173] req@ffff8800937db800 x1800988647394112/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557189 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 599.420554] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557173/real 1717557173] req@ffff8800937dbb80 x1800988647394176/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557189 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 606.043122] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will failLustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 606.158593] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to completeLustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 608.626300] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnectsLustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 609.511839] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 secLustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 23:13:21 (1717557201) [ 612.826955] LustreError: 25382:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 613.162488] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557187/real 1717557187] req@ffff88012b862300 x1800988647399104/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557203 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworLustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557187/real 1717557187] req@ffff88012b860380 x1800988647399168/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557203 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 613.553705] Lustre: Failing over lustre-MDT0000Lustre: server umount lustre-MDT0000 complete [ 618.185556] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557192/real 1717557192] req@ffff88012b861180 x1800988647399360/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557208 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworLustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557192/real 1717557192] req@ffff88012b863480 x1800988647399296/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557208 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworLustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 625.386836] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to completeLustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 627.826718] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 628.373780] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 628.735291] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 631.340981] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 632.065614] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 23:13:40 (1717557220) [ 632.301486] LustreError: 27415:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 632.409505] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 632.413504] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557206/real 1717557206] req@ffff88012f39f480 x1800988647404224/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557222 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 632.807953] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557206/real 1717557206] req@ffff88012f39fb80 x1800988647404288/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557222 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 632.937579] Lustre: Failing over lustre-MDT0000 [ 637.410576] Lustre: server umount lustre-MDT0000 complete [ 637.417516] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557211/real 1717557211] req@ffff88012f39ea00 x1800988647404480/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557227 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 644.490706] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557211/real 1717557211] req@ffff88012f39f100 x1800988647404416/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557227 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 644.590908] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 644.594844] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 645.367257] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.077152] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 647.605466] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 647.961770] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 650.521842] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 650.609485] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 23:13:59 (1717557239) [ 650.609495] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557225/real 1717557225] req@ffff8800a8160a80 x1800988647409344/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557241 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 651.258027] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557225/real 1717557225] req@ffff8800a8163480 x1800988647409280/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557241 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 651.489137] LustreError: 29623:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 651.980858] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 652.074013] Lustre: Failing over lustre-MDT0000 [ 655.617520] Lustre: server umount lustre-MDT0000 complete [ 655.617538] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557230/real 1717557230] req@ffff8800a8161180 x1800988647409472/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557246 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 663.662734] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557230/real 1717557230] req@ffff8800a8163b80 x1800988647409536/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557246 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 663.776773] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 663.777075] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 664.292133] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 664.560824] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 666.739685] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 667.093890] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 669.674868] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 670.411375] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 23:14:19 (1717557259) [ 670.634373] LustreError: 31782:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 670.806534] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 670.809472] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557245/real 1717557245] req@ffff88012b861180 x1800988647414400/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557261 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 671.152730] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557245/real 1717557245] req@ffff88012b863100 x1800988647414336/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557261 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 671.266865] Lustre: Failing over lustre-MDT0000 [ 675.817602] Lustre: server umount lustre-MDT0000 complete [ 675.817628] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557250/real 1717557250] req@ffff88012b860380 x1800988647414592/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557266 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 682.847023] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557250/real 1717557250] req@ffff88012b861880 x1800988647414528/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557266 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 682.964887] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 682.964944] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 683.755945] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 684.324549] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 686.007911] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 686.378346] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 689.041235] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 689.795558] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 23:14:38 (1717557278) [ 689.977562] LustreError: 1455:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 689.977566] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557264/real 1717557264] req@ffff8800a81e1180 x1800988647419392/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557280 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 690.022817] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557264/real 1717557264] req@ffff8800a81e1880 x1800988647419456/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557280 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 690.526891] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 690.637746] Lustre: Failing over lustre-MDT0000 [ 695.001507] Lustre: server umount lustre-MDT0000 complete [ 695.001544] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557269/real 1717557269] req@ffff8800a81e2a00 x1800988647419584/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557285 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 702.219638] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557269/real 1717557269] req@ffff8800a81e3100 x1800988647419648/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557285 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 702.339415] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 702.339422] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 703.054730] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 704.356388] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 705.500420] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 706.029002] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 709.094007] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 709.363599] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 23:14:58 (1717557298) [ 709.372092] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557283/real 1717557283] req@ffff88012b860e00 x1800988647424512/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557299 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 709.941196] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557283/real 1717557283] req@ffff88012b863b80 x1800988647424448/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557299 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 710.181582] LustreError: 3583:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 710.658664] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 710.758471] Lustre: Failing over lustre-MDT0000 [ 713.658660] Lustre: server umount lustre-MDT0000 complete [ 714.377592] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557288/real 1717557288] req@ffff88012b860700 x1800988647424640/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557304 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 722.331457] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557288/real 1717557288] req@ffff88012b862d80 x1800988647424704/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557304 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 722.393673] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 722.393713] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 722.393860] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 722.406265] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 723.215360] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 724.401242] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 725.462798] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 725.813671] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 728.385497] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 728.408065] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557302/real 1717557302] req@ffff880131d59880 x1800988647429888/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557318 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 728.465494] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 23:15:17 (1717557317) [ 729.157979] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557302/real 1717557302] req@ffff880131d58e00 x1800988647429952/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557318 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 729.407825] LustreError: 5761:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 729.921842] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 730.039097] Lustre: Failing over lustre-MDT0000 [ 733.489546] Lustre: server umount lustre-MDT0000 complete [ 733.489554] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557307/real 1717557307] req@ffff880131d5b800 x1800988647430080/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557323 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 741.570408] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557307/real 1717557307] req@ffff880131d58000 x1800988647430144/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557323 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 741.677887] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 741.677925] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 742.416777] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 744.139190] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 744.724688] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 745.082868] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 747.666393] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 747.697520] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 23:15:37 (1717557337) [ 747.701501] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557322/real 1717557322] req@ffff88012c3a7800 x1800988647435392/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557338 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 748.393321] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557322/real 1717557322] req@ffff88012c3a7480 x1800988647435456/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557338 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 748.637655] LustreError: 7996:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 749.128831] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 749.222509] Lustre: Failing over lustre-MDT0000 [ 752.705543] Lustre: server umount lustre-MDT0000 complete [ 753.127611] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557327/real 1717557327] req@ffff88009d55e680 x1800988647435648/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557343 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 760.753420] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557327/real 1717557327] req@ffff88009d55dc00 x1800988647435584/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557343 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 760.848371] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 760.848386] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 761.596916] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 763.296212] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 763.842210] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 764.196148] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 766.738896] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 767.432391] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 23:15:56 (1717557356) [ 767.660369] LustreError: 10169:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 767.865499] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 767.881518] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557342/real 1717557342] req@ffff88009d55df80 x1800988647440640/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557358 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 768.153052] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557342/real 1717557342] req@ffff88009d55ce00 x1800988647440576/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557358 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 768.277261] Lustre: Failing over lustre-MDT0000 [ 772.889581] Lustre: server umount lustre-MDT0000 complete [ 772.889585] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557347/real 1717557347] req@ffff88009d55ea00 x1800988647440768/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557363 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 779.831015] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557347/real 1717557347] req@ffff88009d55e300 x1800988647440832/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557363 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 779.943365] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 779.943367] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 780.705094] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 782.406883] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 782.971303] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 783.343946] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 785.960581] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 786.681823] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 23:16:15 (1717557375) [ 786.906817] LustreError: 12433:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 786.954486] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 787.424962] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557361/real 1717557361] req@ffff880093e1a300 x1800988647445632/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557377 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 787.542714] Lustre: Failing over lustre-MDT0000 [ 788.424566] Lustre: server umount lustre-MDT0000 complete [ 791.977548] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557361/real 1717557361] req@ffff880093e1b800 x1800988647445696/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557377 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 792.424556] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557366/real 1717557366] req@ffff88009de85880 x1800988647445824/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557382 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 799.102357] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557366/real 1717557366] req@ffff88009de86300 x1800988647445888/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557382 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 799.215056] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 799.215071] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 799.515744] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 799.989605] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 802.197705] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 802.561101] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 805.115959] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 805.829705] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 23:16:34 (1717557394) [ 806.044126] LustreError: 14578:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 806.233522] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 806.236809] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557380/real 1717557380] req@ffff88009d55e300 x1800988647450944/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557396 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 806.557789] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557380/real 1717557380] req@ffff88009d55d180 x1800988647451008/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557396 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 806.670206] Lustre: Failing over lustre-MDT0000 [ 811.257604] Lustre: server umount lustre-MDT0000 complete [ 811.257610] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557385/real 1717557385] req@ffff88009d55dc00 x1800988647451200/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557401 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 818.217388] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557385/real 1717557385] req@ffff88009d55d880 x1800988647451136/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557401 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 818.329613] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 818.329665] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 819.105318] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 819.559800] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 821.350725] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 821.714937] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 824.308358] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 825.036424] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 23:16:53 (1717557413) [ 825.272134] LustreError: 16797:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 825.337540] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 825.353505] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557399/real 1717557399] req@ffff880094003480 x1800988647456256/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557415 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 825.774801] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557399/real 1717557399] req@ffff880094000e00 x1800988647456320/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557415 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 825.882715] Lustre: Failing over lustre-MDT0000 [ 830.361733] Lustre: server umount lustre-MDT0000 complete [ 830.361745] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557404/real 1717557404] req@ffff880094003b80 x1800988647456512/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557420 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 837.734006] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557404/real 1717557404] req@ffff880094002d80 x1800988647456448/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557420 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 837.884345] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 837.884348] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.685509] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 839.572651] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 840.992910] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 841.360795] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 843.889531] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 843.893471] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557418/real 1717557418] req@ffff88009ad5a300 x1800988647461504/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557434 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 843.936552] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557418/real 1717557418] req@ffff88009ad5b100 x1800988647461440/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557434 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 844.870461] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 23:17:13 (1717557433) [ 846.817788] Lustre: 19039:0:(/home/green/git/lustre-release/lustre/obdclass/genops.c:1676:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a372da69-b38f-485d-a31f-21d31dc1fbfe at adminstrative request [ 846.930312] Lustre: Failing over lustre-MDT0000 [ 848.817664] Lustre: server umount lustre-MDT0000 complete [ 848.913653] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557423/real 1717557423] req@ffff88009ad58000 x1800988647461632/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557439 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 858.679831] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557423/real 1717557423] req@ffff880091ff3b80 x1800988647461696/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557439 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 858.797645] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 858.797654] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 859.610038] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 859.618602] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 862.062855] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 862.466830] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 863.797497] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 863.804485] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557438/real 1717557438] req@ffff8800a6cece00 x1800988647466880/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557454 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 864.160789] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557438/real 1717557438] req@ffff8800a6ced500 x1800988647466816/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557454 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 868.817501] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 868.817523] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557443/real 1717557443] req@ffff8800a6ced880 x1800988647467072/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557459 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 870.665244] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557443/real 1717557443] req@ffff8800a6cec700 x1800988647467008/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557459 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 872.026010] Lustre: DEBUG MARKER: before 6144, after 6144 [ 872.284854] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 23:17:41 (1717557461) [ 873.829578] Lustre: 21587:0:(/home/green/git/lustre-release/lustre/obdclass/genops.c:1676:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a372da69-b38f-485d-a31f-21d31dc1fbfe at adminstrative request [ 873.830528] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557448/real 1717557448] req@ffff8800953bb800 x1800988647467200/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557464 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 876.643852] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557448/real 1717557448] req@ffff880093cfc000 x1800988647467264/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557464 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 877.413312] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 23:17:46 (1717557466) [ 877.648831] LustreError: 22064:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 878.270033] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 878.412340] Lustre: Failing over lustre-MDT0000 [ 890.558884] Lustre: server umount lustre-MDT0000 complete [ 890.662527] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 890.663461] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 891.597634] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 893.345400] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 893.912352] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 894.267843] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 894.686567] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 894.691102] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557469/real 1717557469] req@ffff880091835500 x1800988647473088/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557485 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 896.994141] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557469/real 1717557469] req@ffff880091835880 x1800988647473024/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557485 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 897.864494] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 23:18:06 (1717557486) [ 898.106259] LustreError: 24149:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 898.650005] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 898.760162] Lustre: Failing over lustre-MDT0000 [ 899.649595] Lustre: server umount lustre-MDT0000 complete [ 899.650041] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557474/real 1717557474] req@ffff880091836680 x1800988647473280/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557490 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 904.649658] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557474/real 1717557474] req@ffff880091836a00 x1800988647473216/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557490 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 904.697574] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557479/real 1717557479] req@ffff880091837100 x1800988647473472/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557495 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 910.404563] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557479/real 1717557479] req@ffff880091837800 x1800988647473408/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557495 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 910.530939] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 910.531203] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 911.672568] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 913.651902] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 914.250894] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 914.785932] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 916.545603] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 916.563569] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557491/real 1717557491] req@ffff880131d05500 x1800988647478400/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557507 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 917.419663] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557491/real 1717557491] req@ffff880131d06300 x1800988647478464/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557507 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 918.487428] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 23:18:26 (1717557506) [ 918.821730] LustreError: 26305:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 919.573302] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 919.724868] Lustre: Failing over lustre-MDT0000 [ 921.569718] Lustre: server umount lustre-MDT0000 complete [ 921.572537] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557496/real 1717557496] req@ffff8800a8161880 x1800988647478656/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557512 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 932.047154] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557496/real 1717557496] req@ffff8800a8162a00 x1800988647478592/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557512 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 932.220991] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 932.221036] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 933.410605] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 934.746180] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 936.208337] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 936.791259] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 937.247563] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 937.267558] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557511/real 1717557511] req@ffff88012f1b2a00 x1800988647483648/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557527 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 939.578492] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557511/real 1717557511] req@ffff88012f1b2d80 x1800988647483712/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557527 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 940.373201] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 23:18:49 (1717557529) [ 940.655625] LustreError: 28514:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 941.367922] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 941.507365] Lustre: Failing over lustre-MDT0000 [ 943.265624] Lustre: server umount lustre-MDT0000 complete [ 943.265632] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557516/real 1717557516] req@ffff88012f1b3800 x1800988647483904/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557532 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 947.265618] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557516/real 1717557516] req@ffff88012f1b1c00 x1800988647483840/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557532 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 948.273706] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557521/real 1717557521] req@ffff8800a7377100 x1800988647484096/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557537 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 953.549224] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557521/real 1717557521] req@ffff88012f1b3b80 x1800988647484032/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557537 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 953.667993] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 953.670554] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 954.780320] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 954.803318] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 957.462030] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 957.676556] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 957.696573] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557532/real 1717557532] req@ffff8800a6cece00 x1800988647489088/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557548 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 957.936533] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557532/real 1717557532] req@ffff8800a6cedf80 x1800988647489152/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557548 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 961.078755] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 962.056170] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 23:19:10 (1717557550) [ 962.394030] LustreError: 30636:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 962.681664] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 962.811609] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557537/real 1717557537] req@ffff8800a6cec000 x1800988647489280/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557553 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 963.176717] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557537/real 1717557537] req@ffff8800a6cee300 x1800988647489344/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557553 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 963.344106] Lustre: Failing over lustre-MDT0000 [ 967.705545] Lustre: server umount lustre-MDT0000 complete [ 968.174793] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557542/real 1717557542] req@ffff8800a6ceed80 x1800988647489536/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557558 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 975.208900] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557542/real 1717557542] req@ffff8800a6ced880 x1800988647489472/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557558 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 975.309838] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 975.314304] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 976.153288] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 977.956092] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 978.806742] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 979.273091] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 980.318536] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 980.340473] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557554/real 1717557554] req@ffff880091837100 x1800988647494528/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557570 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 982.599654] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557554/real 1717557554] req@ffff880091835500 x1800988647494592/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557570 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 983.704074] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 23:19:31 (1717557571) [ 984.046236] LustreError: 330:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 984.839930] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 984.982594] Lustre: Failing over lustre-MDT0000 [ 984.999515] Lustre: server umount lustre-MDT0000 complete [ 985.860635] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557559/real 1717557559] req@ffff880091834380 x1800988647494784/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557575 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 989.839688] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557559/real 1717557559] req@ffff880091837480 x1800988647494720/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557575 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 989.853866] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557564/real 1717557564] req@ffff880094389c00 x1800988647494976/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557580 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 996.895671] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557564/real 1717557564] req@ffff880094389880 x1800988647494912/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557580 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 997.026564] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 997.026604] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 997.933220] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 999.852076] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1000.495087] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1001.011074] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1001.039518] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1001.055532] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557575/real 1717557575] req@ffff880091835880 x1800988647500032/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557591 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1004.155923] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557575/real 1717557575] req@ffff880091836d80 x1800988647500096/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557591 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1005.124602] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 23:19:53 (1717557593) [ 1005.462554] LustreError: 2530:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1006.041524] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1006.057538] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557580/real 1717557580] req@ffff88009d4a3100 x1800988647500224/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557596 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1006.240649] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557580/real 1717557580] req@ffff88009d4a2300 x1800988647500288/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557596 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1006.390398] Lustre: Failing over lustre-MDT0000 [ 1011.065668] Lustre: server umount lustre-MDT0000 complete [ 1011.240737] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557585/real 1717557585] req@ffff88009de84000 x1800988647500480/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557601 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1018.531559] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557585/real 1717557585] req@ffff88009de86a00 x1800988647500416/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557601 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1018.654164] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1018.654178] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1019.758715] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1019.896793] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1022.653542] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1022.682047] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557597/real 1717557597] req@ffff880131d5b800 x1800988647505536/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557613 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1022.700612] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1023.270682] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557597/real 1717557597] req@ffff880131d5b480 x1800988647505600/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557613 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1026.350079] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1027.473714] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 23:20:15 (1717557615) [ 1027.806138] LustreError: 4859:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1027.963704] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1028.081647] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557602/real 1717557602] req@ffff880131d58e00 x1800988647505792/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1028.590238] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557602/real 1717557602] req@ffff880131d59f80 x1800988647505728/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1028.697719] Lustre: Failing over lustre-MDT0000 [ 1032.588637] Lustre: server umount lustre-MDT0000 complete [ 1032.613608] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557607/real 1717557607] req@ffff88009d4a2300 x1800988647505920/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557623 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1040.930839] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557607/real 1717557607] req@ffff88009d4a3100 x1800988647505984/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557623 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1041.096167] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1041.096171] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1042.307793] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1044.335651] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1045.114573] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1045.119019] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557619/real 1717557619] req@ffff88009ce2df80 x1800988647511232/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557635 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1045.306624] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557619/real 1717557619] req@ffff88009ce2c000 x1800988647511168/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557635 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1045.868135] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1048.616526] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1049.505394] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 23:20:37 (1717557637) [ 1049.737404] LustreError: 7023:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1050.121481] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1050.223621] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557624/real 1717557624] req@ffff88009d7a1500 x1800988647511424/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557640 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1050.328762] Lustre: Failing over lustre-MDT0000 [ 1050.411591] Lustre: server umount lustre-MDT0000 complete [ 1055.145607] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557624/real 1717557624] req@ffff8800a6986d80 x1800988647511360/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557640 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1055.145630] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557629/real 1717557629] req@ffff88009d7a3b80 x1800988647511616/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557645 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1062.769032] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557629/real 1717557629] req@ffff88009d7a3480 x1800988647511552/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557645 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1062.929337] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1062.929348] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1064.133949] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1064.960100] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1066.648792] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1066.950517] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1066.980540] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557641/real 1717557641] req@ffff880131d59f80 x1800988647516736/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557657 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1067.068921] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557641/real 1717557641] req@ffff880131d58e00 x1800988647516800/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557657 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1069.867571] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1070.650962] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 23:20:59 (1717557659) [ 1070.911132] LustreError: 9272:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1071.490681] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1071.594039] Lustre: Failing over lustre-MDT0000 [ 1072.490562] Lustre: server umount lustre-MDT0000 complete [ 1072.490566] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557646/real 1717557646] req@ffff880131d05c00 x1800988647516992/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557662 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1076.969651] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557646/real 1717557646] req@ffff880131d05500 x1800988647516928/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557662 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1077.490640] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557651/real 1717557651] req@ffff880131d06300 x1800988647517120/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557667 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1083.866584] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557651/real 1717557651] req@ffff880131d05180 x1800988647517184/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557667 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1084.025094] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1084.025112] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1084.965834] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1085.138230] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1088.025822] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1088.653251] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1089.037587] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1089.042536] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557663/real 1717557663] req@ffff880091836a00 x1800988647522368/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557679 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1092.058490] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557663/real 1717557663] req@ffff880091834700 x1800988647522304/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557679 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1093.247385] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 23:21:21 (1717557681) [ 1093.606513] LustreError: 11505:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1094.059569] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1094.357894] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557668/real 1717557668] req@ffff880091837800 x1800988647522560/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557684 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1094.498995] Lustre: Failing over lustre-MDT0000 [ 1095.073571] Lustre: server umount lustre-MDT0000 complete [ 1099.076645] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557668/real 1717557668] req@ffff880091837480 x1800988647522496/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557684 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1100.081618] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557673/real 1717557673] req@ffff88009d55ca80 x1800988647522752/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557689 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1106.899697] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557673/real 1717557673] req@ffff88009d55e300 x1800988647522688/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557689 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1107.040201] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1107.044821] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1108.095894] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1110.005631] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1110.970855] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1111.536267] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1115.037828] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1115.042610] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 23:21:44 (1717557704) [ 1115.073589] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557689/real 1717557689] req@ffff880091835880 x1800988647528320/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557705 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1115.430986] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557689/real 1717557689] req@ffff880091836d80 x1800988647528384/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557705 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1119.946645] Lustre: 13638:0:(/home/green/git/lustre-release/lustre/obdclass/genops.c:1676:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a372da69-b38f-485d-a31f-21d31dc1fbfe at adminstrative request [ 1120.081559] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 23:21:49 (1717557709) [ 1120.081566] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557694/real 1717557694] req@ffff880091836a00 x1800988647528512/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557710 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1120.645729] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557694/real 1717557694] req@ffff880091837b80 x1800988647528576/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557710 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1120.964729] LustreError: 14087:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1121.590077] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1121.724613] Lustre: Failing over lustre-MDT0000 [ 1124.471949] Lustre: server umount lustre-MDT0000 complete [ 1124.618267] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1124.618308] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.687028] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.693032] Lustre: lustre-MDT0000: Aborting client recovery [ 1124.695358] LustreError: 14975:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2927:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1124.702180] Lustre: 15115:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1124.708884] Lustre: 15115:0:(/home/green/git/lustre-release/lustre/obdclass/genops.c:1533:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a372da69-b38f-485d-a31f-21d31dc1fbfe@ [ 1124.717366] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1124.719933] Lustre: 15115:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1125.878828] Lustre: 15115:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1131.287194] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1132.120184] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 23:22:00 (1717557720) [ 1132.395684] LustreError: 16235:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1133.029802] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1133.139682] Lustre: Failing over lustre-MDT0000 [ 1135.336421] Lustre: server umount lustre-MDT0000 complete [ 1135.437312] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1135.438518] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1135.498498] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1135.501525] Lustre: lustre-MDT0000: Aborting client recovery [ 1135.502700] LustreError: 17131:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2927:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1135.506171] Lustre: 17365:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1135.509480] Lustre: 17365:0:(/home/green/git/lustre-release/lustre/obdclass/genops.c:1533:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a372da69-b38f-485d-a31f-21d31dc1fbfe@ [ 1135.514129] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1135.515507] Lustre: 17365:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1136.473932] Lustre: 17365:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1138.440661] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1138.441561] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557712/real 1717557712] req@ffff8800a6c14a80 x1800988647534080/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557728 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1141.685140] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557712/real 1717557712] req@ffff8800a6c14000 x1800988647534144/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557728 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1142.729607] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 23:22:11 (1717557731) [ 1143.069070] LustreError: 18420:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1143.780795] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1143.896448] Lustre: Failing over lustre-MDT0000 [ 1146.687600] Lustre: server umount lustre-MDT0000 complete [ 1146.820178] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1146.820197] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.914398] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.919191] Lustre: lustre-MDT0000: Aborting client recovery [ 1146.921033] LustreError: 19355:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2927:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1146.927373] Lustre: 19602:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1146.933551] Lustre: 19602:0:(/home/green/git/lustre-release/lustre/obdclass/genops.c:1533:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a372da69-b38f-485d-a31f-21d31dc1fbfe@ [ 1146.940561] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1146.942654] Lustre: 19602:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1148.040366] Lustre: 19602:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1150.870582] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1150.875403] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557725/real 1717557725] req@ffff88009de84a80 x1800988647539264/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557741 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1153.464968] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557725/real 1717557725] req@ffff88009de86a00 x1800988647539328/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557741 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1153.863806] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 23:22:22 (1717557742) [ 1156.551400] LustreError: 19362:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff88009b0eb100 x1800988641423808/t201863462916(0) o36->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:705/0 lens 512/456 e 0 to 0 dl 1717557755 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 [ 1156.694219] Lustre: Failing over lustre-MDT0000 [ 1159.474895] Lustre: server umount lustre-MDT0000 complete [ 1159.613059] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1159.613143] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.673281] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.679047] Lustre: lustre-MDT0000: Aborting client recovery [ 1159.681225] LustreError: 21346:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2927:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1159.688025] Lustre: 21503:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1159.695080] Lustre: 21503:0:(/home/green/git/lustre-release/lustre/obdclass/genops.c:1533:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a372da69-b38f-485d-a31f-21d31dc1fbfe@ [ 1159.702687] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1159.705063] Lustre: 21503:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1160.624523] Lustre: 21503:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1160.628510] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557735/real 1717557735] req@ffff88009b3d3100 x1800988647546880/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557751 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1160.751591] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557735/real 1717557735] req@ffff88009b3d1500 x1800988647546944/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557751 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1166.232416] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1166.824697] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 1167.922982] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 23:22:36 (1717557756) [ 1168.263549] LustreError: 22579:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1169.326652] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1169.447323] Lustre: Failing over lustre-MDT0000 [ 1172.028236] Lustre: server umount lustre-MDT0000 complete [ 1172.148905] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1172.148908] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1172.211430] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1172.217416] Lustre: lustre-MDT0000: Aborting client recovery [ 1172.219526] LustreError: 23573:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2927:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1172.226575] Lustre: 23703:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1172.234025] Lustre: 23703:0:(/home/green/git/lustre-release/lustre/obdclass/genops.c:1533:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a372da69-b38f-485d-a31f-21d31dc1fbfe@ [ 1172.243530] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1172.246028] Lustre: 23703:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1172.657537] Lustre: 23703:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1172.665426] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557747/real 1717557747] req@ffff88009d55ca80 x1800988647554432/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557763 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1173.403392] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557747/real 1717557747] req@ffff88009d55d180 x1800988647554368/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557763 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1179.261283] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1186.185508] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 23:22:48 (1717557768) [ 1186.185513] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557760/real 1717557760] req@ffff8800a8161500 x1800988647559424/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557776 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1188.415340] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557760/real 1717557760] req@ffff8800a8160a80 x1800988647559488/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557776 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1188.720841] LustreError: 25502:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1189.441654] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1189.555723] Lustre: Failing over lustre-MDT0000 [ 1201.820683] Lustre: server umount lustre-MDT0000 complete [ 1201.928072] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1201.932767] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1202.857380] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1204.727599] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1205.564408] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1206.093647] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1207.953652] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1207.994563] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557782/real 1717557782] req@ffff88009af84700 x1800988647584256/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1212.961572] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557782/real 1717557782] req@ffff88009af85c00 x1800988647584320/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1212.961638] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557787/real 1717557787] req@ffff88009af87480 x1800988647584512/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557803 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1214.605611] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557787/real 1717557787] req@ffff88009af85500 x1800988647584448/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557803 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1220.525318] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 23:23:23 (1717557803) [ 1220.866047] LustreError: 28615:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1224.436037] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1224.464201] Lustre: Failing over lustre-MDT0000 [ 1224.577610] LustreError: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1281:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800a740a300 x1800988647643456/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 [ 1236.623225] Lustre: server umount lustre-MDT0000 complete [ 1236.745268] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1236.745585] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.858562] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1239.853596] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1241.697749] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1242.293965] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1242.753558] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1242.753566] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557817/real 1717557817] req@ffff8800a13bdf80 x1800988647645760/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557833 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1247.777564] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557817/real 1717557817] req@ffff8800a13be300 x1800988647645696/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557833 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1247.790500] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557822/real 1717557822] req@ffff8800a13bf100 x1800988647645888/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557838 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1251.077490] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557822/real 1717557822] req@ffff8800a13bd180 x1800988647645952/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557838 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1251.703741] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 23:24:00 (1717557840) [ 1252.086284] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1252.091186] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1252.098960] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1254.133764] LustreError: 167-0: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1260.267135] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 23:24:03 (1717557843) [ 1260.606764] LustreError: 32632:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only *** [ 1264.369738] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 1264.394056] Lustre: Failing over lustre-OST0000 [ 1265.257613] Lustre: server umount lustre-OST0000 complete [ 1265.818080] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1265.826850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1270.260212] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1270.826087] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1275.268135] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1275.833876] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.290953] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.467843] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1324.044477] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1325.176230] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 23:25:13 (1717557913) [ 1325.499949] LustreError: 3340:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1326.564662] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1326.671845] Lustre: Failing over lustre-MDT0000 [ 1338.518685] Lustre: server umount lustre-MDT0000 complete [ 1338.622445] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1338.622470] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1339.760604] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.384539] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1342.653516] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1342.665513] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1342.665522] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557917/real 1717557917] req@ffff8801353faa00 x1800988647762048/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557933 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1343.139231] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557917/real 1717557917] req@ffff8801353fb800 x1800988647761984/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557933 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1347.657617] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1347.665535] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557922/real 1717557922] req@ffff8801353f8000 x1800988647762176/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557938 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1352.665564] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557922/real 1717557922] req@ffff8801353fa300 x1800988647762240/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557938 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1352.665569] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557927/real 1717557927] req@ffff88009cd0d180 x1800988647762432/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557943 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1356.432602] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557927/real 1717557927] req@ffff8801353fad80 x1800988647762368/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557943 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1356.448178] Lustre: 4376:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557930/real 1717557930] req@ffff88009d6d9180 x1800988647777664/t0(0) o5->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1717557946 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre [ 1356.463431] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1356.464109] LustreError: 4376:0:(/home/green/git/lustre-release/lustre/osp/osp_precreate.c:992:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 1356.650943] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1358.475048] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 23:25:45 (1717557945) [ 1363.481601] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1363.488626] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1364.215555] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1369.218627] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1369.225344] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1369.963141] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1374.968599] LustreError: 4374:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1374.978722] LustreError: 4374:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1375.689108] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1380.693620] LustreError: 4374:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1380.700763] LustreError: 4374:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1381.426750] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1386.432676] LustreError: 4374:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1386.440838] LustreError: 4374:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1387.132032] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1392.136585] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1392.140339] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1392.812253] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1397.818654] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1397.825151] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1398.521493] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1403.526695] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1403.532866] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1404.232001] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1409.238679] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1409.245798] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1409.956583] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1414.961551] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1414.968217] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1417.980504] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1418.549920] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 23:26:47 (1717558007) [ 1418.556919] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1428.550562] LustreError: 4331:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1433.556505] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1438.564107] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1443.572100] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1448.580103] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1453.588254] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1458.561591] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1458.566600] LustreError: 4331:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake [ 1458.596142] Lustre: 4331:0:(/home/green/git/lustre-release/lustre/ptlrpc/service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff88012c3a4380 x1800988642732224/t0(0) o38->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:0/0 lens 520/416 e 0 to 0 dl 1717558029 ref 1 fl Complete:H/200/0 rc 0/0 job:'lc [ 1459.314762] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1459.319639] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1469.315508] LustreError: 4331:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1474.324401] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1479.332058] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1480.024858] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1484.340327] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1489.348406] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1494.356368] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1499.323580] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1499.328535] LustreError: 4331:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake [ 1499.364017] Lustre: 4331:0:(/home/green/git/lustre-release/lustre/ptlrpc/service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8800953bb800 x1800988642734272/t0(0) o38->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:0/0 lens 520/416 e 0 to 0 dl 1717558069 ref 1 fl Complete:H/200/0 rc 0/0 job:'lc [ 1499.380393] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1524.364510] LustreError: 4331:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1529.380060] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1534.388136] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1539.385542] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1539.390512] LustreError: 4331:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake [ 1539.398200] Lustre: 4331:0:(/home/green/git/lustre-release/lustre/ptlrpc/service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8800a8162680 x1800988642735424/t0(0) o38->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:0/0 lens 520/416 e 0 to 0 dl 1717558109 ref 1 fl Complete:H/200/0 rc 0/0 job:'kw [ 1539.413218] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1564.396830] LustreError: 4331:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1569.412329] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1574.420193] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1579.417599] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1579.423749] LustreError: 4331:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake [ 1579.427856] Lustre: 4331:0:(/home/green/git/lustre-release/lustre/ptlrpc/service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff88009ce2fb80 x1800988642736320/t0(0) o38->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:0/0 lens 520/416 e 0 to 0 dl 1717558149 ref 1 fl Complete:H/200/0 rc 0/0 job:'kw [ 1579.427866] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1604.428189] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1609.444093] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1614.452349] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1619.460547] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1619.527562] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1619.534558] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake [ 1624.467732] Lustre: 4332:0:(/home/green/git/lustre-release/lustre/ptlrpc/service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/21s); client may timeout req@ffff88009b3d1500 x1800988642737216/t0(0) o38->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:0/0 lens 520/416 e 0 to 0 dl 1717558189 ref 1 fl Complete:H/200/0 rc 0/0 job:'kw [ 1624.479601] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1645.282517] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1645.291742] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:148:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 1646.809590] Lustre: 4332:0:(/home/green/git/lustre-release/lustre/ptlrpc/service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/1s); client may timeout req@ffff88012b0f2680 x1800988642738304/t0(0) o38->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:0/0 lens 520/416 e 0 to 0 dl 1717558234 ref 1 fl Complete:H/200/0 rc 0/0 job:'kwo [ 1647.790645] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 23:30:36 (1717558236) [ 1648.132227] LustreError: 9524:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1649.623640] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1649.768696] Lustre: Failing over lustre-MDT0000 [ 1652.208486] Lustre: server umount lustre-MDT0000 complete [ 1652.300719] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1652.342805] LustreError: 19025:0:(/home/green/git/lustre-release/lustre/ptlrpc/service.c:1236:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff880131d58a80 x1800988647810560/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 [ 1652.353282] Lustre: lustre-MDT0000: Aborting client recovery [ 1652.355388] LustreError: 10624:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2927:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1652.359552] Lustre: 10763:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1652.363576] Lustre: 10763:0:(/home/green/git/lustre-release/lustre/obdclass/genops.c:1533:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a372da69-b38f-485d-a31f-21d31dc1fbfe@ [ 1652.368632] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1652.370211] Lustre: 10763:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1653.472249] Lustre: 10763:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1657.322007] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1657.322168] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1657.553742] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1657.681845] Lustre: Failing over lustre-MDT0000 [ 1670.147271] Lustre: server umount lustre-MDT0000 complete [ 1670.312562] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1670.312580] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1671.480623] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1673.546345] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1674.414844] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1675.003957] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1678.307845] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1678.337541] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 23:31:07 (1717558267) [ 1678.337552] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558252/real 1717558252] req@ffff8800a0765180 x1800988647814912/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558268 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1678.361428] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558252/real 1717558252] req@ffff8800a0764700 x1800988647814976/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558268 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1678.440151] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1681.609876] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1681.963025] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 23:31:10 (1717558270) [ 1683.329598] LustreError: 12193:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff8800a6c65180 x1800988642763392/t0(0) o700->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:478/0 lens 264/248 e 0 to 0 dl 1717558283 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 [ 1683.345695] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558257/real 1717558257] req@ffff8800a0764000 x1800988647815104/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558273 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1697.985787] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558257/real 1717558257] req@ffff8800a0765880 x1800988647815168/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558273 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1699.078769] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1699.219007] Lustre: Failing over lustre-MDT0000 [ 1711.642259] Lustre: server umount lustre-MDT0000 complete [ 1711.792092] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1711.792121] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1712.793575] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1712.799560] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558242/real 1717558242] req@ffff8800a0766a00 x1800988647809920/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558303 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1712.989997] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558242/real 1717558242] req@ffff8800a0766d80 x1800988647809984/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558303 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1714.045048] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1715.791607] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1715.840545] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558290/real 1717558290] req@ffff8800a7375180 x1800988647821888/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558306 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1715.909482] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558290/real 1717558290] req@ffff8800a7377800 x1800988647821824/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558306 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1716.513360] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1719.297819] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1720.188774] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 23:31:48 (1717558308) [ 1720.211093] Lustre: Failing over lustre-OST0000 [ 1721.188629] Lustre: server umount lustre-OST0000 complete [ 1721.817975] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558295/real 1717558295] req@ffff8800a7376680 x1800988647822016/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558311 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1721.835865] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1721.840813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1724.052302] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1725.833571] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1726.841842] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558300/real 1717558300] req@ffff8800a6c65880 x1800988647822208/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558316 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1729.044007] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1731.849768] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1734.052192] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1734.314198] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1736.968994] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1737.529642] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1802.646940] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 1803.817396] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 23:33:11 (1717558391) [ 1804.143883] LustreError: 17865:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1804.980222] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1805.112040] Lustre: Failing over lustre-MDT0000 [ 1817.601012] Lustre: server umount lustre-MDT0000 complete [ 1817.705870] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1817.705917] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1818.932270] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1819.189587] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1823.793600] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1823.793618] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558398/real 1717558398] req@ffff88012d4ab100 x1800988647838784/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558414 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1828.801629] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558398/real 1717558398] req@ffff880094aef480 x1800988647838720/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558414 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1828.801640] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558403/real 1717558403] req@ffff88009b3d3480 x1800988647838976/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558419 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1883.870872] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558403/real 1717558403] req@ffff88009b3d1c00 x1800988647838912/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558419 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1884.594549] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 23:34:33 (1717558473) [ 1884.604082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1884.953864] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1884.959132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1892.151836] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1893.205035] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 23:34:41 (1717558481) [ 1893.376789] Lustre: Failing over lustre-MDT0000 [ 1905.247472] Lustre: server umount lustre-MDT0000 complete [ 1905.367697] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1905.367763] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1906.565380] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.664311] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1908.672974] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1914.381497] LustreError: 21733:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff88009de84a80 x1800988642818240/t0(0) o101->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:705/0 lens 328/344 e 0 to 0 dl 1717558510 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 [ 1914.387690] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558488/real 1717558488] req@ffff8801353fad80 x1800988647854400/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558504 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1919.401572] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558488/real 1717558488] req@ffff8801353f8000 x1800988647854336/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558504 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1919.401611] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558493/real 1717558493] req@ffff8801353f9c00 x1800988647854528/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558509 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1924.355978] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558493/real 1717558493] req@ffff8801353faa00 x1800988647854592/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558509 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1924.860207] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnected, waiting for 1 clients in recovery for 0:54 [ 1925.205227] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1929.081027] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1931.863533] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 23:35:18 (1717558518) [ 1932.190605] LustreError: 23165:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1932.912245] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1933.058273] Lustre: Failing over lustre-MDT0000 [ 1945.229126] Lustre: server umount lustre-MDT0000 complete [ 1945.359624] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1945.359648] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.498022] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.500654] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1949.447052] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1949.973088] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1951.361555] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1951.377511] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558525/real 1717558525] req@ffff8800a8161f80 x1800988647861312/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558541 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1953.607806] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558525/real 1717558525] req@ffff8800a8162a00 x1800988647861376/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558541 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1955.984972] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 23:35:42 (1717558542) [ 1956.247032] LustreError: 25576:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1956.364530] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1956.385531] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558530/real 1717558530] req@ffff8800a8162680 x1800988647861568/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558546 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1956.926261] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558530/real 1717558530] req@ffff8800a8161180 x1800988647861504/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558546 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1957.093407] Lustre: Failing over lustre-MDT0000 [ 1969.660498] Lustre: server umount lustre-MDT0000 complete [ 1969.808565] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1969.808780] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1970.068588] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1970.931669] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1973.817091] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1974.396175] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1975.825512] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1975.835493] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558550/real 1717558550] req@ffff88012c3a6a00 x1800988647867136/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558566 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1977.670978] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558550/real 1717558550] req@ffff88012c3a5f80 x1800988647867200/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558566 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1980.444689] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 23:36:06 (1717558566) [ 1980.731247] LustreError: 27851:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1980.833553] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1981.100524] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558555/real 1717558555] req@ffff88012c3a5180 x1800988647867392/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558571 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1981.414622] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558555/real 1717558555] req@ffff88012c3a7480 x1800988647867328/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558571 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1981.576179] Lustre: Failing over lustre-MDT0000 [ 1994.089774] Lustre: server umount lustre-MDT0000 complete [ 1994.127210] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1994.238793] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1994.247349] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1995.182951] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1995.455842] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2000.498075] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2001.273638] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 23:36:29 (1717558589) [ 2001.273646] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558575/real 1717558575] req@ffff8800953ff800 x1800988647872960/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558591 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2001.889617] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558575/real 1717558575] req@ffff8800953fea00 x1800988647873024/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558591 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2002.916410] LustreError: 28712:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff88009b8c2a00 x1800988642839488/t257698037777(0) o35->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:43/0 lens 392/456 e 0 to 0 dl 1717558603 ref 1 fl Interpret:/200/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 2003.049571] Lustre: Failing over lustre-MDT0000 [ 2005.916676] Lustre: server umount lustre-MDT0000 complete [ 2006.281665] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558580/real 1717558580] req@ffff8800953fdf80 x1800988647873152/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558596 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2015.261678] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558580/real 1717558580] req@ffff880094aef480 x1800988647873216/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558596 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2015.420966] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2015.424186] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2016.476673] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2017.920894] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2017.954663] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2019.236309] Lustre: 30703:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800a12d4000 x1800988642839488/t257698037777(0) o35->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:59/0 lens 392/456 e 0 to 0 dl 1717558619 ref 1 fl Interpret:/202/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 2019.736376] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2020.441573] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2021.457577] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558594/real 1717558594] req@ffff8800a13bd500 x1800988647878336/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558610 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2023.474309] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558594/real 1717558594] req@ffff8800a13bdf80 x1800988647878272/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558610 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2023.876205] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 23:36:52 (1717558612) [ 2025.457607] LustreError: 30701:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff8800a13be300 x1800988642845504/t261993005072(0) o36->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:65/0 lens 504/448 e 0 to 0 dl 1717558625 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 2026.054895] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558599/real 1717558599] req@ffff8800a13bf100 x1800988647878464/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558615 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2026.402188] LustreError: 32187:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2026.465550] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2027.153247] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558599/real 1717558599] req@ffff8800a13bc000 x1800988647878528/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558615 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2027.293604] Lustre: Failing over lustre-MDT0000 [ 2030.152577] Lustre: server umount lustre-MDT0000 complete [ 2030.152637] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558604/real 1717558604] req@ffff88009ce2fb80 x1800988647878720/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558620 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2039.695847] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558604/real 1717558604] req@ffff8800a13bed80 x1800988647878656/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558620 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2039.846907] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2039.847001] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2040.996990] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2041.064812] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2041.099092] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2043.623037] Lustre: 593:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff880131d59500 x1800988642845504/t261993005072(0) o36->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:82/0 lens 504/2880 e 0 to 0 dl 1717558642 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 2044.228287] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2045.862599] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2045.873523] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558620/real 1717558620] req@ffff88012c3a4e00 x1800988647884608/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558636 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2047.850013] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558620/real 1717558620] req@ffff88012c3a5f80 x1800988647884544/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558636 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2048.261870] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 23:37:17 (1717558637) [ 2049.602797] LustreError: 594:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff8800a7374380 x1800988642851584/t266287972368(0) o36->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:89/0 lens 504/448 e 0 to 0 dl 1717558649 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 2050.657275] LustreError: 597:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff8800a7375880 x1800988642851712/t266287972369(0) o35->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:91/0 lens 392/456 e 0 to 0 dl 1717558651 ref 1 fl Interpret:/200/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 2050.881554] LustreError: 2085:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2050.881570] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558625/real 1717558625] req@ffff88009ce2df80 x1800988647884800/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558641 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2051.024874] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558625/real 1717558625] req@ffff88009ce2fb80 x1800988647884736/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558641 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2051.687659] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2051.840592] Lustre: Failing over lustre-MDT0000 [ 2063.943151] Lustre: server umount lustre-MDT0000 complete [ 2064.057118] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2064.061613] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.288769] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.335308] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2064.335341] Lustre: 2988:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800a7375180 x1800988642851584/t266287972368(0) o36->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:105/0 lens 504/2880 e 0 to 0 dl 1717558665 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 2065.259341] Lustre: 2946:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800a7374700 x1800988642851712/t266287972369(0) o35->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:105/0 lens 392/456 e 0 to 0 dl 1717558665 ref 1 fl Interpret:/202/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 2070.788261] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2071.081574] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 23:37:39 (1717558659) [ 2071.196389] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558645/real 1717558645] req@ffff88012f1b2d80 x1800988647890176/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558661 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2071.327629] LustreError: 2943:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff8800a7377480 x1800988642857280/t270582939664(0) o36->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:112/0 lens 504/448 e 0 to 0 dl 1717558672 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 2073.868667] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558645/real 1717558645] req@ffff88012f1b1180 x1800988647890240/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558661 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2074.181234] LustreError: 4429:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2074.867622] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2075.015218] Lustre: Failing over lustre-MDT0000 [ 2075.867588] Lustre: server umount lustre-MDT0000 complete [ 2075.867652] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558650/real 1717558650] req@ffff88012f1b2a00 x1800988647890368/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558666 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2087.414852] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558650/real 1717558650] req@ffff880094389c00 x1800988647890432/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558666 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2087.531996] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2087.543698] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2088.697360] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2094.532618] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2094.569598] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558669/real 1717558669] req@ffff88012f1b3b80 x1800988647895744/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558685 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2096.254025] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558669/real 1717558669] req@ffff88012f1b1f80 x1800988647895808/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558685 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2096.297715] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2099.577643] Lustre: 5337:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88009de87100 x1800988642857280/t270582939664(0) o36->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:137/0 lens 504/2880 e 0 to 0 dl 1717558697 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 2099.577651] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558674/real 1717558674] req@ffff8800a6986300 x1800988647896000/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558690 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2100.664277] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558674/real 1717558674] req@ffff8800a6985500 x1800988647895936/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558690 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2102.384328] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 23:38:09 (1717558689) [ 2104.350361] LustreError: 5340:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff88009d55c000 x1800988642863232/t274877906960(0) o35->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:143/0 lens 392/456 e 0 to 0 dl 1717558703 ref 1 fl Interpret:/200/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 2104.689137] LustreError: 6748:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2105.347729] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2105.476673] Lustre: Failing over lustre-MDT0000 [ 2117.773695] Lustre: server umount lustre-MDT0000 complete [ 2118.925823] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2122.938280] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2122.946275] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.922840] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.930320] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558698/real 1717558698] req@ffff88009d55ea00 x1800988647902016/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558714 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2127.126030] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558698/real 1717558698] req@ffff88009d55ed80 x1800988647902080/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558714 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2127.167633] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2128.945558] Lustre: 7643:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88009de86300 x1800988642863232/t274877906960(0) o35->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:168/0 lens 392/456 e 0 to 0 dl 1717558728 ref 1 fl Interpret:/202/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 2128.945567] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558703/real 1717558703] req@ffff88009d55f480 x1800988647902272/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558719 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2131.748329] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558703/real 1717558703] req@ffff88009d55c000 x1800988647902208/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558719 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2132.059535] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 23:38:41 (1717558721) [ 2133.953606] LustreError: 8204:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff88009de86a00 x1800988642867584/t279172874255(0) o101->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:173/0 lens 664/608 e 0 to 0 dl 1717558733 ref 1 fl Interpret:/200/0 rc 301/0 job:'touch.0' uid:0 gid:0 [ 2133.953612] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558708/real 1717558708] req@ffff88009d55f100 x1800988647902464/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558724 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2148.082207] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558708/real 1717558708] req@ffff88009d55f800 x1800988647902400/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558724 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2148.100082] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 2150.328414] Lustre: 8204:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800a6ced180 x1800988642867584/t279172874255(0) o101->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:189/0 lens 664/3488 e 0 to 0 dl 1717558749 ref 1 fl Interpret:/202/0 rc 0/0 job:'touch.0' uid:0 gid:0 [ 2151.403611] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 23:38:59 (1717558739) [ 2151.698070] LustreError: 9416:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2152.404040] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2152.534199] Lustre: Failing over lustre-MDT0000 [ 2165.016710] Lustre: server umount lustre-MDT0000 complete [ 2165.170262] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2165.170301] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2166.389841] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.118179] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2169.190533] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2169.196265] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558743/real 1717558743] req@ffff8800a7723800 x1800988647910080/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558759 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2169.285429] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558743/real 1717558743] req@ffff8800a7720380 x1800988647910016/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558759 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2169.814069] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2174.179591] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2174.185525] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558748/real 1717558748] req@ffff8800a7722300 x1800988647910208/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558764 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2179.209550] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558748/real 1717558748] req@ffff8800a7720e00 x1800988647910272/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558764 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2179.209577] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558753/real 1717558753] req@ffff8800a7722680 x1800988647910400/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558769 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2182.420272] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558753/real 1717558753] req@ffff8800a7722d80 x1800988647910464/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558769 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2183.616003] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 23:39:31 (1717558771) [ 2183.834243] LustreError: 11787:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2184.304802] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2184.393919] Lustre: Failing over lustre-MDT0000 [ 2195.919799] Lustre: server umount lustre-MDT0000 complete [ 2196.018266] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2196.018268] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2196.730582] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2198.179713] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2198.911421] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2199.251980] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2200.916867] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2201.029491] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2201.043524] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558775/real 1717558775] req@ffff880093e19c00 x1800988647916032/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558791 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2205.423256] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558775/real 1717558775] req@ffff880093e1bb80 x1800988647916096/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558791 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2206.023510] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 23:39:54 (1717558794) [ 2206.033582] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558780/real 1717558780] req@ffff88009a274700 x1800988647916224/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558796 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2211.041493] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558780/real 1717558780] req@ffff88009a276680 x1800988647916288/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558796 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2211.041499] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558785/real 1717558785] req@ffff88009a277480 x1800988647916416/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558801 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2212.496008] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558785/real 1717558785] req@ffff88009a274e00 x1800988647916480/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558801 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2212.733310] LustreError: 14867:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2213.204712] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2213.341297] Lustre: Failing over lustre-MDT0000 [ 2224.895792] Lustre: server umount lustre-MDT0000 complete [ 2224.997436] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2224.997857] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2225.747157] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2227.436472] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2227.996655] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2228.337574] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2231.999478] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2232.025485] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558806/real 1717558806] req@ffff8800a0863b80 x1800988647927360/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558822 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2237.017512] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558806/real 1717558806] req@ffff8800a0860a80 x1800988647927424/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558822 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2237.033496] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558811/real 1717558811] req@ffff8800a0862680 x1800988647927552/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558827 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2241.449346] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558811/real 1717558811] req@ffff8800a0860700 x1800988647927616/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558827 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2242.312486] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 23:40:30 (1717558830) [ 2242.671626] LustreError: 17704:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2243.165124] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2243.282528] Lustre: Failing over lustre-MDT0000 [ 2254.799945] Lustre: server umount lustre-MDT0000 complete [ 2254.924874] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2254.924949] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2255.708032] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2256.564647] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2257.901323] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2258.225407] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2260.266676] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2260.611835] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 2260.945487] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 2260.945492] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558835/real 1717558835] req@ffff8800a0942680 x1800988648187392/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558851 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2261.962036] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558835/real 1717558835] req@ffff8800a0943800 x1800988648187328/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558851 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2265.953549] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 23:40:51 (1717558851) [ 2265.953556] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558840/real 1717558840] req@ffff88009a8ff800 x1800988648187584/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558856 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2283.372384] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558840/real 1717558840] req@ffff88009a8fce00 x1800988648187520/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558856 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2283.885380] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 2299.893473] LustreError: 18764:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff88008f240a80 x1800988644335360/t296352743433(0) o36->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:325/0 lens 66040/440 e 0 to 0 dl 1717558885 ref 1 fl Interpret:/200/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 [ 2299.907269] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 2301.833357] Lustre: 20085:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88009b8c1180 x1800988644335360/t296352743433(0) o36->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:341/0 lens 66040/440 e 0 to 0 dl 1717558901 ref 1 fl Interpret:/202/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 [ 2302.195106] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 2304.054960] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 23:41:31 (1717558891) [ 2304.273225] LustreError: 21395:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2304.875795] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2304.964123] Lustre: Failing over lustre-MDT0000 [ 2317.079934] Lustre: server umount lustre-MDT0000 complete [ 2317.228872] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2317.229125] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2318.397267] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2321.237642] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2321.282633] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558895/real 1717558895] req@ffff88009b0eaa00 x1800988648390272/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558911 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2325.932531] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558895/real 1717558895] req@ffff88009b0e9c00 x1800988648390336/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558911 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2326.244551] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2327.085051] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558900/real 1717558900] req@ffff88009b0ebb80 x1800988648390528/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558916 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2327.256592] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2327.669394] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558900/real 1717558900] req@ffff88009b0eb480 x1800988648390464/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558916 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2331.590118] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2332.239602] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 23:42:00 (1717558920) [ 2332.273597] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558905/real 1717558905] req@ffff88009b0ea680 x1800988648390656/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558921 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2336.816752] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558905/real 1717558905] req@ffff88009b0ead80 x1800988648390720/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558921 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2337.028007] LustreError: 24046:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only *** [ 2340.910698] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 2340.932186] Lustre: Failing over lustre-OST0000 [ 2341.246926] Lustre: server umount lustre-OST0000 complete [ 2341.247310] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2341.247348] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2341.247430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2341.247473] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2341.247631] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2341.247818] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2341.249703] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2341.287744] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2341.291136] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2342.297677] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2344.970286] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2347.306092] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2349.972149] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.313875] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2354.442802] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2355.032331] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2366.455779] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2366.475922] Lustre: Failing over lustre-OST0000 [ 2368.298198] Lustre: server umount lustre-OST0000 complete [ 2368.301216] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2369.021934] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2373.305958] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.020200] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2378.313806] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2379.027694] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.617282] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2383.496251] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2384.087103] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2417.897689] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2418.918644] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 23:43:27 (1717559007) [ 2419.067011] Lustre: Failing over lustre-MDT0000 [ 2431.539170] Lustre: server umount lustre-MDT0000 complete [ 2431.685661] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2431.689593] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2432.810794] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2434.140047] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2439.690687] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2439.713584] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559014/real 1717559014] req@ffff880131d5bb80 x1800988648474624/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559030 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2444.181952] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559014/real 1717559014] req@ffff880131d59180 x1800988648474688/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559030 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2444.323609] Lustre: Failing over lustre-MDT0000 [ 2444.721566] Lustre: server umount lustre-MDT0000 complete [ 2445.181587] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559019/real 1717559019] req@ffff880131d5b800 x1800988648474816/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559035 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2456.590069] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559019/real 1717559019] req@ffff880131d58000 x1800988648474880/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559035 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2456.746031] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2456.746062] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2457.917389] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2459.187398] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2460.866154] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2461.471196] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2462.769582] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2462.785589] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559037/real 1717559037] req@ffff880091835180 x1800988648480128/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559053 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2464.996930] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559037/real 1717559037] req@ffff880091837800 x1800988648480064/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559053 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2467.793604] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 23:44:14 (1717559054) [ 2467.801415] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559042/real 1717559042] req@ffff8800a0893100 x1800988648480256/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559058 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2476.257815] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559042/real 1717559042] req@ffff8800a0893800 x1800988648480320/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559058 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2476.287876] Lustre: Failing over lustre-OST0000 [ 2476.793915] Lustre: server umount lustre-OST0000 complete [ 2476.796595] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2476.801474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.210899] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.817860] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2484.196026] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2486.825828] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.203790] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2490.320028] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2493.198605] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2493.791178] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2497.750390] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2498.469878] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 23:44:46 (1717559086) [ 2498.592899] Lustre: Failing over lustre-MDT0000 [ 2501.725831] Lustre: server umount lustre-MDT0000 complete [ 2501.729017] LustreError: 2255:0:(/home/green/git/lustre-release/lustre/obdclass/llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffa0451350 failed: rc = -95 [ 2501.736668] LustreError: 2255:0:(/home/green/git/lustre-release/lustre/obdclass/obd_config.c:797:class_setup()) setup MGS failed (-95) [ 2501.743341] LustreError: 2255:0:(/home/green/git/lustre-release/lustre/obdclass/obd_mount.c:216:lustre_start_simple()) MGS setup error -95 [ 2501.749973] LustreError: 2255:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/tgt_mount.c:135:server_deregister_mount()) MGS not registered [ 2501.756839] LustreError: 0-0: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 2501.761212] LustreError: 2255:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/tgt_mount.c:1751:server_put_super()) no obd lustre-MDT0000 [ 2501.779651] LustreError: 2255:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/tgt_mount.c:135:server_deregister_mount()) lustre-MDT0000 not registered [ 2501.785509] Lustre: server umount lustre-MDT0000 complete [ 2503.814882] LustreError: 2255:0:(/home/green/git/lustre-release/lustre/llite/super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 2504.257198] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2505.138276] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2508.041971] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2508.986015] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 23:44:57 (1717559097) [ 2508.986086] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.149978] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.486265] LustreError: 3728:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2510.677249] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2510.813403] Lustre: Failing over lustre-MDT0000 [ 2514.676626] Lustre: server umount lustre-MDT0000 complete [ 2514.679356] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559089/real 1717559089] req@ffff8801353f8a80 x1800988648489856/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559105 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2519.696596] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559089/real 1717559089] req@ffff8801353fad80 x1800988648489792/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559105 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2520.001592] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559094/real 1717559094] req@ffff8801353fa300 x1800988648490048/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559110 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2523.371298] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559094/real 1717559094] req@ffff8801353f8000 x1800988648489984/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559110 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2523.545894] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2523.554141] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2524.291164] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2524.755290] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2529.569540] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2529.569570] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559104/real 1717559104] req@ffff8800a8163800 x1800988648494720/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2534.577580] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559104/real 1717559104] req@ffff8800a8160a80 x1800988648494784/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2534.577647] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559109/real 1717559109] req@ffff8800a8160e00 x1800988648494912/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559125 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2540.296170] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559109/real 1717559109] req@ffff8800a8161f80 x1800988648494976/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559125 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2541.565919] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnected, waiting for 1 clients in recovery for 0:53 [ 2542.138990] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2546.201972] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2569.638258] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 23:45:35 (1717559135) [ 2580.642632] LustreError: 5271:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 11000ms [ 2581.770997] LustreError: 5271:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2585.687670] Lustre: DEBUG MARKER: replay-single test_65a: @@@@@@ FAIL: No early reply [ 2609.197143] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 23:46:14 (1717559174) [ 2620.204567] LustreError: 14520:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 sleeping for 11000ms [ 2621.092574] LustreError: 14520:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 awake [ 2625.418850] Lustre: DEBUG MARKER: replay-single test_65b: @@@@@@ FAIL: No early reply [ 2648.532755] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 23:46:54 (1717559214) [ 2653.546573] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 5000ms [ 2654.416130] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2664.422541] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 10000ms [ 2676.351912] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2771.016596] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 23:47:45 (1717559265) [ 2794.008024] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 23:49:20 (1717559360) [ 2794.415553] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2794.424357] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2794.829545] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2794.839005] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2795.244542] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2795.252902] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2795.658582] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2795.667523] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2796.072557] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2796.081624] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2796.169791] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2796.169805] LustreError: 8600:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2796.486548] LustreError: 12604:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2796.494681] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2796.581542] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2796.669536] LustreError: 8600:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2796.900595] LustreError: 12604:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2796.909523] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2797.314495] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2797.319556] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2797.721495] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2797.726507] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2798.129644] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2798.138602] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2798.543568] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2798.553540] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2798.962567] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2798.973404] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2799.378586] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2799.387640] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2799.793546] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2799.802747] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2800.208652] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2800.217269] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2800.623576] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2800.632643] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2801.037555] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2801.046647] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2801.451555] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2801.460027] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2801.594011] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2801.689931] LustreError: 12604:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2801.865549] LustreError: 8600:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2801.875665] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2802.001552] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2802.096592] LustreError: 12604:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2802.284547] LustreError: 8600:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2802.292354] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2802.697533] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2802.706079] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2803.111539] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2803.119619] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2803.523536] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2803.530089] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2803.932512] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2803.938115] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2804.340551] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2804.347615] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2804.750497] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2804.756637] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2805.159658] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2805.168434] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2805.573571] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2805.582880] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2805.988528] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2805.997930] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2806.403501] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2806.408427] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2806.813560] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2806.822421] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2807.017951] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2807.113807] LustreError: 8600:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2807.227555] LustreError: 12604:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2807.236705] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2807.420537] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2807.516529] LustreError: 8600:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2807.642539] LustreError: 12604:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2807.653223] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2808.058555] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2808.066939] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2808.472617] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2808.482345] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2808.887549] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2808.896296] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2809.301547] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2809.311713] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2809.716559] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2809.725671] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2810.131657] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2810.141543] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2810.546611] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2810.555531] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2810.960588] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2810.969940] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2811.377580] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2811.386252] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2811.791530] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2811.799852] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2812.205601] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2812.214612] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2812.441829] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2812.537904] LustreError: 12604:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2812.619541] LustreError: 8600:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2812.628032] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2812.847537] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2812.945543] LustreError: 12604:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2813.035548] LustreError: 8600:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2813.043417] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2813.448540] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2813.457279] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2813.863564] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2813.871733] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2814.276537] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2814.285119] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2814.690549] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2814.699571] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2815.105545] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2815.113509] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2815.518551] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2815.526774] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2815.932547] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2815.941341] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2816.346553] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2816.354407] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2816.759547] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2816.768061] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2817.173550] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2817.182947] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2817.588594] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2817.597129] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2817.865948] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2817.961894] LustreError: 8600:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2818.002547] LustreError: 12604:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2818.013859] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2818.271574] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2818.367549] LustreError: 8600:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2818.423545] LustreError: 12604:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2818.436294] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2818.841549] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2818.853725] LustreError: 10550:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2819.259552] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2819.268383] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2819.673576] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2819.684082] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2820.091595] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2820.103538] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2820.508554] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2820.517815] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2820.923569] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2820.932560] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2821.337548] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2821.346743] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2821.752553] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2821.761087] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2822.166569] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2822.175644] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2822.584551] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2822.593289] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2822.999556] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2823.008252] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2823.289852] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2823.385818] LustreError: 12604:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2823.413535] LustreError: 8600:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2823.421894] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2823.482401] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2823.482458] LustreError: 8597:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2823.482465] LustreError: 16743:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2823.482470] LustreError: 27570:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2823.482482] LustreError: 19024:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2823.695532] LustreError: 16737:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2823.794542] LustreError: 12604:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2823.827541] LustreError: 8600:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2823.836418] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2823.882518] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2823.882528] LustreError: 16737:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2823.882539] LustreError: 27570:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2823.882550] LustreError: 16743:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2823.912527] LustreError: 19024:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2824.241585] LustreError: 8597:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2824.249954] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2824.655590] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2824.666805] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2825.073522] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2825.082102] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2825.487547] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2825.496553] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2825.903565] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2825.912520] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2826.318556] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2826.327794] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2826.733557] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2826.742592] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2827.148638] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2827.162803] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2827.567561] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2827.576581] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2827.981560] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2827.990541] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2828.395565] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2828.404517] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2828.481033] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2828.481042] LustreError: 27570:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2828.481047] LustreError: 16737:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2828.481052] LustreError: 16743:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2828.481063] LustreError: 16749:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2828.481066] LustreError: 8597:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2828.713876] LustreError: 16747:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2828.809537] LustreError: 8600:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2828.809868] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2828.825750] LustreError: 12604:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2828.924554] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2828.980484] LustreError: 27570:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2828.980528] LustreError: 16743:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2828.980539] LustreError: 16749:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2828.980556] LustreError: 8597:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2828.981060] LustreError: 16737:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2829.122543] LustreError: 16747:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2829.233548] LustreError: 8600:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2829.241696] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2829.309567] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2829.646550] LustreError: 12604:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2829.655238] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2830.060532] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2830.069337] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2830.475503] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2830.480494] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2830.882548] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2830.887421] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2831.292499] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2831.301523] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2831.703481] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2831.708269] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2832.113504] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2832.118343] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2832.520536] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2832.529104] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2832.934584] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2832.949072] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2833.354524] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2833.359542] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2833.480928] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2833.480932] LustreError: 8597:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2833.480939] LustreError: 16743:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2833.481026] LustreError: 16737:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2833.481048] LustreError: 27570:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2833.481053] LustreError: 10477:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2833.765548] LustreError: 16747:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2833.774860] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2833.917550] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2833.980540] LustreError: 8597:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2833.980555] LustreError: 16747:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2833.980566] LustreError: 16737:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2833.980573] LustreError: 10477:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2833.980800] LustreError: 27570:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2834.137746] LustreError: 16743:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2834.180544] LustreError: 12604:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2834.188418] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2834.329864] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2834.542604] LustreError: 8600:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2834.593552] LustreError: 12604:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2834.602095] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2834.735544] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2835.006564] LustreError: 8600:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2835.014675] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2835.419561] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 2837.305508] LustreError: 4688:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2860.993512] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 23:50:26 (1717559426) [ 2863.231858] Lustre: DEBUG MARKER: phase 2 [ 2932.327106] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 23:50:52 (1717559452) [ 2932.869966] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 23:52:01 (1717559521) [ 2933.512107] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 2935.257289] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 23:52:02 (1717559522) [ 2937.410000] Lustre: DEBUG MARKER: Started rundbench load pid=4305 ... [ 2937.761954] LustreError: 17776:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2939.310916] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2939.990653] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 2940.121670] Lustre: Failing over lustre-MDT0000 [ 2952.508550] Lustre: server umount lustre-MDT0000 complete [ 2952.647663] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2952.647716] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2953.813647] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2959.673572] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2959.673660] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559534/real 1717559534] req@ffff8800953bad80 x1800988648587136/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559550 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2964.673619] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559534/real 1717559534] req@ffff8800953b9500 x1800988648587072/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559550 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2964.681625] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559539/real 1717559539] req@ffff8800953b9180 x1800988648587328/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559555 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2969.982401] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559539/real 1717559539] req@ffff8800953ba680 x1800988648587264/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559555 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2971.556897] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2972.120383] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2975.378043] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2975.675747] LustreError: 21787:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2977.110954] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2977.811748] Lustre: DEBUG MARKER: test_70b fail mds1 2 times [ 2977.941747] Lustre: Failing over lustre-MDT0000 [ 2990.327685] Lustre: server umount lustre-MDT0000 complete [ 2990.481398] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2990.481407] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2991.293635] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2995.004660] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2996.652579] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2997.235082] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2999.310531] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2999.310550] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559573/real 1717559573] req@ffff88009de85c00 x1800988648645184/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559589 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3000.675851] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559573/real 1717559573] req@ffff88009de85f80 x1800988648645120/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559589 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3001.039044] LustreError: 25273:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3002.614330] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3003.325404] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 3003.457833] Lustre: Failing over lustre-MDT0000 [ 3004.344577] Lustre: server umount lustre-MDT0000 complete [ 3004.345554] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559578/real 1717559578] req@ffff88009f3c4380 x1800988648645312/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559594 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3015.917356] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559578/real 1717559578] req@ffff88009f3c7480 x1800988648645376/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559594 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3017.200140] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3020.028209] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3021.081816] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3021.082004] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.713176] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3022.269088] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3024.784508] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3024.784564] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559596/real 1717559596] req@ffff88009f0d5180 x1800988648703808/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3025.647903] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559596/real 1717559596] req@ffff88009f0d6300 x1800988648703872/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3025.913819] LustreError: 29450:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3027.422289] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3028.039502] Lustre: DEBUG MARKER: test_70b fail mds1 4 times [ 3028.039547] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559601/real 1717559601] req@ffff88009f0d7480 x1800988648704064/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3028.117731] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559601/real 1717559601] req@ffff88009f0d7800 x1800988648704000/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3028.177645] Lustre: Failing over lustre-MDT0000 [ 3028.274847] LustreError: 27983:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:1499:ldlm_handle_enqueue()) ### lock on destroyed export ffff880129d40800 ns: mdt-lustre-MDT0000_UUID lock: ffff88012daf9d40/0xbb24dbe123918a90 lrc: 3/0,0 mode: PR/PR res: [0x20001a9e3:0x1085:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.203.45@t [ 3032.097622] Lustre: server umount lustre-MDT0000 complete [ 3032.097630] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559606/real 1717559606] req@ffff88009f0d5880 x1800988648704256/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3040.441572] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559606/real 1717559606] req@ffff88009f0d6680 x1800988648704192/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3040.611723] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3040.611915] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3041.664718] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3045.052671] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3046.831899] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3047.341343] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3047.434483] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3047.434568] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559621/real 1717559621] req@ffff88009d039180 x1800988648779392/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3050.675623] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559621/real 1717559621] req@ffff88009d038e00 x1800988648779328/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3051.009406] LustreError: 1431:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3051.674611] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3051.677521] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559626/real 1717559626] req@ffff88009d03a300 x1800988648779520/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559642 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3052.460778] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559626/real 1717559626] req@ffff88009d038380 x1800988648779584/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559642 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3053.003837] Lustre: DEBUG MARKER: test_70b fail mds1 5 times [ 3053.120532] Lustre: Failing over lustre-MDT0000 [ 3065.512839] Lustre: server umount lustre-MDT0000 complete [ 3065.646688] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3065.646695] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3066.341596] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3070.076506] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3071.693533] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3072.140390] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559646/real 1717559646] req@ffff88012c3a7480 x1800988648850176/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559662 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3072.301563] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3072.752203] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559646/real 1717559646] req@ffff88012c3a4e00 x1800988648850240/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559662 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3076.262418] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3076.607228] LustreError: 5635:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3076.721502] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3077.313547] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559651/real 1717559651] req@ffff88012c3a5500 x1800988648850432/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559667 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3078.181931] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559651/real 1717559651] req@ffff88012c3a5180 x1800988648850368/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559667 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3078.876594] Lustre: DEBUG MARKER: test_70b fail mds1 6 times [ 3079.005375] Lustre: Failing over lustre-MDT0000 [ 3091.342654] Lustre: server umount lustre-MDT0000 complete [ 3091.475964] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3091.480834] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3092.625785] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3095.100710] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3097.280871] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3097.788509] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3097.885953] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559671/real 1717559671] req@ffff88009c868000 x1800988648910336/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559687 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3098.844547] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3101.421631] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559671/real 1717559671] req@ffff88009c869880 x1800988648910272/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559687 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3101.736233] LustreError: 9326:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3103.288300] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3103.464509] Lustre: DEBUG MARKER: test_70b fail mds1 7 times [ 3103.468247] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559676/real 1717559676] req@ffff88009c869c00 x1800988648910464/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559692 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3103.991775] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559676/real 1717559676] req@ffff88009c86ad80 x1800988648910528/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559692 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3104.138344] Lustre: Failing over lustre-MDT0000 [ 3107.025656] Lustre: server umount lustre-MDT0000 complete [ 3107.028611] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559681/real 1717559681] req@ffff8800a0851c00 x1800988648910656/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559697 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3116.585204] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559681/real 1717559681] req@ffff8800a0850e00 x1800988648910720/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559697 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3117.851408] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3120.124614] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3121.737826] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3121.737925] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3121.964851] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3122.571280] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3122.753553] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3122.759929] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559697/real 1717559697] req@ffff8800a7285500 x1800988648980096/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559713 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3125.996720] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559697/real 1717559697] req@ffff8800a7285c00 x1800988648980160/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559713 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3126.353208] LustreError: 13545:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3127.745551] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3127.920590] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559702/real 1717559702] req@ffff88009b3d1500 x1800988648980288/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559718 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3128.637692] Lustre: DEBUG MARKER: test_70b fail mds1 8 times [ 3128.755189] Lustre: Failing over lustre-MDT0000 [ 3129.637579] Lustre: server umount lustre-MDT0000 complete [ 3132.635628] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559702/real 1717559702] req@ffff88009b3d2680 x1800988648980352/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559718 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3132.753578] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559707/real 1717559707] req@ffff880130c40000 x1800988648980480/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559723 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3141.083758] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559707/real 1717559707] req@ffff880130c40e00 x1800988648980544/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559723 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3141.237608] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3141.237631] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3142.357070] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3145.147813] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3146.692589] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3147.296135] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3148.249557] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3148.265545] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559722/real 1717559722] req@ffff88009b298700 x1800988649042112/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559738 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3150.768295] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559722/real 1717559722] req@ffff88009b298380 x1800988649042048/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559738 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3151.153132] LustreError: 17579:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3152.736593] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3153.435248] Lustre: DEBUG MARKER: test_70b fail mds1 9 times [ 3153.552922] Lustre: Failing over lustre-MDT0000 [ 3154.447579] Lustre: server umount lustre-MDT0000 complete [ 3154.447589] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559727/real 1717559727] req@ffff88009b29a300 x1800988649042240/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559743 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3166.002470] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559727/real 1717559727] req@ffff88009b299c00 x1800988649042304/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559743 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3166.142827] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3166.142831] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3167.286659] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3170.172126] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3171.665191] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3172.235963] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3173.147508] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3173.147524] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559746/real 1717559746] req@ffff88009a5b8000 x1800988649106752/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559762 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3175.563682] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559746/real 1717559746] req@ffff88009a5b8700 x1800988649106688/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559762 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3175.842959] LustreError: 21356:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3177.185511] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3177.185566] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559751/real 1717559751] req@ffff88009a5b8e00 x1800988649106880/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559767 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3177.415863] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559751/real 1717559751] req@ffff88009a5b8a80 x1800988649106944/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559767 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3178.121114] Lustre: DEBUG MARKER: test_70b fail mds1 10 times [ 3178.237422] Lustre: Failing over lustre-MDT0000 [ 3190.728376] Lustre: server umount lustre-MDT0000 complete [ 3190.893746] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3190.893751] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3192.067564] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3195.196304] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3196.887514] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3196.913482] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3196.965559] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559771/real 1717559771] req@ffff880090252a00 x1800988649165120/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559787 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3197.240192] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559771/real 1717559771] req@ffff880090251880 x1800988649165184/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559787 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3200.648501] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3200.967736] LustreError: 25085:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3202.521489] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3203.009556] Lustre: DEBUG MARKER: test_70b fail mds1 11 times [ 3203.012258] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559776/real 1717559776] req@ffff8800a0b68e00 x1800988649165376/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559792 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3203.217787] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559776/real 1717559776] req@ffff8800a0b6ad80 x1800988649165312/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559792 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3203.260492] Lustre: Failing over lustre-MDT0000 [ 3203.347690] LustreError: 23150:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:1499:ldlm_handle_enqueue()) ### lock on destroyed export ffff880136bcd000 ns: mdt-lustre-MDT0000_UUID lock: ffff880099c5c900/0xbb24dbe12395cf93 lrc: 3/0,0 mode: PR/PR res: [0x20001a9e3:0xee2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.203.45@tc [ 3215.754974] Lustre: server umount lustre-MDT0000 complete [ 3215.905015] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3215.905038] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3217.045084] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3220.220280] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3221.879812] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3222.455377] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3223.514516] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3225.376523] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559796/real 1717559796] req@ffff880099e01f80 x1800988649231872/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559812 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3225.588765] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559796/real 1717559796] req@ffff880099e01180 x1800988649231808/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559812 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3225.887332] LustreError: 29489:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3227.221977] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3227.672654] Lustre: DEBUG MARKER: test_70b fail mds1 12 times [ 3227.757745] Lustre: Failing over lustre-MDT0000 [ 3228.672614] Lustre: server umount lustre-MDT0000 complete [ 3228.675934] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559801/real 1717559801] req@ffff880099e02680 x1800988649232064/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559817 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3239.191486] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559801/real 1717559801] req@ffff880099e00000 x1800988649232000/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559817 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3239.279386] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3239.279388] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3240.057309] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3244.267550] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3246.143718] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3246.620956] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3247.986498] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3247.986502] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559821/real 1717559821] req@ffff880129db0e00 x1800988649311552/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559837 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3252.672586] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559821/real 1717559821] req@ffff880129db1c00 x1800988649311616/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559837 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3252.698274] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559826/real 1717559826] req@ffff880129db0000 x1800988649311744/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559842 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3297.223942] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559826/real 1717559826] req@ffff880129db0380 x1800988649311808/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559842 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3418.115064] Lustre: DEBUG MARKER: == replay-single test 70c: tar 1mdts recovery ============ 23:58:06 (1717559886) [ 3418.361821] LustreError: 25376:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3428.720034] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3429.194926] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 3429.345783] Lustre: Failing over lustre-MDT0000 [ 3440.941541] Lustre: server umount lustre-MDT0000 complete [ 3441.042753] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3441.042782] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3441.877172] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3446.115551] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3447.073568] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560020/real 1717560020] req@ffff880134edd500 x1800988650957632/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560036 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3449.603698] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560020/real 1717560020] req@ffff880134eddf80 x1800988650957696/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560036 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3451.074471] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3452.161488] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560025/real 1717560025] req@ffff880134edea00 x1800988650957824/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560041 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3457.211494] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560025/real 1717560025] req@ffff880134edd180 x1800988650957888/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560041 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3457.211501] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560030/real 1717560030] req@ffff8800877a8a80 x1800988650958016/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560046 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3459.298983] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560030/real 1717560030] req@ffff8800877a8e00 x1800988650958080/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560046 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3459.699222] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3581.801999] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3582.092160] LustreError: 4480:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3592.458211] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3592.959839] Lustre: DEBUG MARKER: test_70c fail mds1 2 times [ 3593.116391] Lustre: Failing over lustre-MDT0000 [ 3604.754869] Lustre: server umount lustre-MDT0000 complete [ 3604.864666] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3604.864668] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3605.701313] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3609.859521] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3611.897495] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3611.952523] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560186/real 1717560186] req@ffff88009c2e6680 x1800988652481152/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560202 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3617.025489] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560186/real 1717560186] req@ffff88009c2e5c00 x1800988652481088/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560202 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3617.025496] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560191/real 1717560191] req@ffff880134edd180 x1800988652481344/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560207 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3620.784248] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560191/real 1717560191] req@ffff880134edce00 x1800988652481280/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560207 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3621.180756] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3643.982839] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3644.302572] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 1mdts recovery ========================================================== 00:03:53 (1717560233) [ 3644.675310] Lustre: DEBUG MARKER: SKIP: replay-single test_70d needs >= 2 MDTs [ 3645.000921] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 00:03:54 (1717560234) [ 3645.405767] Lustre: DEBUG MARKER: SKIP: replay-single test_70e needs >= 2 MDTs [ 3649.663251] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 00:03:54 (1717560234) [ 3649.948764] LustreError: 11891:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only *** [ 3651.471736] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3652.130037] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 3652.162899] Lustre: Failing over lustre-OST0000 [ 3654.934032] Lustre: server umount lustre-OST0000 complete [ 3654.970039] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3654.973645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3659.939751] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3659.977809] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3664.947512] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3665.066388] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3667.545825] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3668.106430] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3675.498563] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3675.828409] LustreError: 18053:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only *** [ 3677.360813] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3678.033804] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 3678.049668] Lustre: Failing over lustre-OST0000 [ 3678.921929] Lustre: server umount lustre-OST0000 complete [ 3678.924786] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3679.976634] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3683.929914] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3684.979970] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3688.937979] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3689.987805] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3690.364650] mount.lustre (19003) used greatest stack depth: 9560 bytes left [ 3691.638913] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3692.068124] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3694.782531] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3695.292097] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3702.667658] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3702.998936] LustreError: 24455:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only *** [ 3704.529293] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3705.142658] Lustre: DEBUG MARKER: test_70f failing OST 3 times [ 3705.158099] Lustre: Failing over lustre-OST0000 [ 3705.369777] Lustre: server umount lustre-OST0000 complete [ 3705.373234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3710.024625] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3710.377835] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3715.028116] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3715.385661] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3718.623890] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3719.329733] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3722.177206] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3722.657243] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3729.682965] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3729.894861] LustreError: 32470:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only *** [ 3731.221012] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3731.680770] Lustre: DEBUG MARKER: test_70f failing OST 4 times [ 3731.690686] Lustre: Failing over lustre-OST0000 [ 3732.521846] Lustre: server umount lustre-OST0000 complete [ 3732.523213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3735.061960] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3737.529914] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3740.067661] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3742.537791] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3744.290394] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3744.930149] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3746.477359] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3746.831125] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3753.739764] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3753.951200] LustreError: 8749:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only *** [ 3755.263177] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3755.700605] Lustre: DEBUG MARKER: test_70f failing OST 5 times [ 3755.710961] Lustre: Failing over lustre-OST0000 [ 3758.201864] Lustre: server umount lustre-OST0000 complete [ 3758.203237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3760.102084] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3763.209918] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3765.107705] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3768.349329] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3768.792601] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3770.527901] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3770.875350] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3775.406114] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3775.708714] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 00:06:04 (1717560364) [ 3776.058095] Lustre: DEBUG MARKER: SKIP: replay-single test_71a needs >= 2 MDTs [ 3776.715805] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 00:06:05 (1717560365) [ 3776.920388] LustreError: 15638:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3777.542695] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3777.616089] Lustre: Failing over lustre-MDT0000 [ 3788.939061] Lustre: server umount lustre-MDT0000 complete [ 3789.029171] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3789.029215] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3789.679609] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3790.162290] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3798.057548] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3798.057553] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560372/real 1717560372] req@ffff88008ad01500 x1800988652985472/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560388 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3803.065542] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560372/real 1717560372] req@ffff88008ad02a00 x1800988652985536/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560388 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3803.065545] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560377/real 1717560377] req@ffff88008ad02680 x1800988652985664/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560393 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3806.176839] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560377/real 1717560377] req@ffff88008ad03100 x1800988652985728/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560393 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3806.654774] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnected, waiting for 1 clients in recovery for 0:53 [ 3806.967132] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3809.493245] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3810.177120] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 00:06:38 (1717560398) [ 3810.389649] LustreError: 17999:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3811.068666] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3811.162953] Lustre: Failing over lustre-MDT0000 [ 3822.535341] Lustre: server umount lustre-MDT0000 complete [ 3822.609637] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3822.612400] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3823.262965] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3824.859379] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3824.865139] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3829.625520] LustreError: 18938:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff88008cd91f80 x1800988680289536/t382252089347(382252089347) o101->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:356/0 lens 592/608 e 0 to 0 dl 1717560426 ref 1 fl Interpret:/204/0 rc 301/0 job:'multiop.0' uid:0 gid:0 [ 3829.637679] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560404/real 1717560404] req@ffff880094b83100 x1800988652992064/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560420 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3834.633511] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560404/real 1717560404] req@ffff880094b83b80 x1800988652992128/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560420 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3834.633518] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560409/real 1717560409] req@ffff880094b82680 x1800988652992320/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560425 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3841.231996] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560409/real 1717560409] req@ffff880094b81880 x1800988652992256/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560425 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3841.242310] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnected, waiting for 1 clients in recovery for 0:53 [ 3841.723654] Lustre: 18939:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff880086844a80 x1800988680289536/t382252089347(382252089347) o101->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:372/0 lens 592/3488 e 0 to 0 dl 1717560442 ref 1 fl Interpret:/206/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 3842.065235] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3844.613812] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3845.165505] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 00:07:14 (1717560434) [ 3845.183284] Lustre: Failing over lustre-OST0000 [ 3846.147052] Lustre: server umount lustre-OST0000 complete [ 3846.156617] Lustre: Failing over lustre-MDT0000 [ 3846.266361] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3857.689991] Lustre: server umount lustre-MDT0000 complete [ 3857.769430] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3858.474888] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3858.996793] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3859.903790] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3860.730391] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3860.733420] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3860.954207] Lustre: lustre-OST0000: Denying connection for new client 231c734d-bd3e-4978-ac09-ae891bc5f7a4 (at 192.168.203.45@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 3863.833500] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3866.585396] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560438/real 1717560438] req@ffff880134492a00 x1800988652998784/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560454 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3866.891997] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 00:07:36 (1717560456) [ 3867.228168] Lustre: DEBUG MARKER: SKIP: replay-single test_80a needs >= 2 MDTs [ 3867.528876] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 00:07:36 (1717560456) [ 3867.853799] Lustre: DEBUG MARKER: SKIP: replay-single test_80b needs >= 2 MDTs [ 3868.142236] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 00:07:37 (1717560457) [ 3868.460311] Lustre: DEBUG MARKER: SKIP: replay-single test_80c needs >= 2 MDTs [ 3868.748995] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 00:07:37 (1717560457) [ 3868.905489] Lustre: DEBUG MARKER: SKIP: replay-single test_80d needs >= 2 MDTs [ 3869.081400] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560443/real 1717560443] req@ffff8800896c1880 x1800988652998912/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560459 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3869.383170] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 00:07:38 (1717560458) [ 3869.727854] Lustre: DEBUG MARKER: SKIP: replay-single test_80e needs >= 2 MDTs [ 3870.008631] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 00:07:39 (1717560459) [ 3870.331812] Lustre: DEBUG MARKER: SKIP: replay-single test_80f needs >= 2 MDTs [ 3870.615631] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 00:07:39 (1717560459) [ 3870.944338] Lustre: DEBUG MARKER: SKIP: replay-single test_80g needs >= 2 MDTs [ 3871.231851] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 00:07:40 (1717560460) [ 3871.570118] Lustre: DEBUG MARKER: SKIP: replay-single test_80h needs >= 2 MDTs [ 3871.869065] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 00:07:41 (1717560461) [ 3872.244970] Lustre: DEBUG MARKER: SKIP: replay-single test_81a needs >= 2 MDTs [ 3872.554681] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 00:07:41 (1717560461) [ 3872.927118] Lustre: DEBUG MARKER: SKIP: replay-single test_81b needs >= 2 MDTs [ 3873.247949] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 00:07:42 (1717560462) [ 3873.594059] Lustre: DEBUG MARKER: SKIP: replay-single test_81c needs >= 2 MDTs [ 3873.857521] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 00:07:43 (1717560463) [ 3873.885925] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560448/real 1717560448] req@ffff8800896c3b80 x1800988652999040/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560464 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3874.223044] Lustre: DEBUG MARKER: SKIP: replay-single test_81d needs >= 2 MDTs [ 3874.522740] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 00:07:43 (1717560463) [ 3874.856789] Lustre: DEBUG MARKER: SKIP: replay-single test_81e needs >= 2 MDTs [ 3875.143519] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 00:07:44 (1717560464) [ 3875.542230] Lustre: DEBUG MARKER: SKIP: replay-single test_81f needs >= 2 MDTs [ 3875.848080] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 00:07:44 (1717560464) [ 3876.193594] Lustre: DEBUG MARKER: SKIP: replay-single test_81g needs >= 2 MDTs [ 3876.478137] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 00:07:45 (1717560465) [ 3876.820089] Lustre: DEBUG MARKER: SKIP: replay-single test_81h needs >= 2 MDTs [ 3877.257057] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 00:07:46 (1717560466) [ 3880.897963] Lustre: 25467:0:(/home/green/git/lustre-release/lustre/obdclass/genops.c:1676:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 231c734d-bd3e-4978-ac09-ae891bc5f7a4 at adminstrative request [ 3881.982820] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 00:07:50 (1717560470) [ 3882.085830] Lustre: Failing over lustre-MDT0000 [ 3893.479572] Lustre: server umount lustre-MDT0000 complete [ 3893.557234] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3893.557370] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3894.052468] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3894.202789] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3896.386924] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3896.743863] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3899.345863] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3900.569480] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 00:08:08 (1717560488) [ 3900.575423] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560475/real 1717560475] req@ffff8800877c5f80 x1800988653007104/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560491 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3901.571656] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560475/real 1717560475] req@ffff8800877c4380 x1800988653007040/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560491 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3901.613721] Lustre: Failing over lustre-OST0000 [ 3903.577734] Lustre: server umount lustre-OST0000 complete [ 3903.577738] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3903.577893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3904.079044] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3905.582546] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3908.585737] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560480/real 1717560480] req@ffff88013083c380 x1800988653007232/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560496 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3909.075618] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3914.054335] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3914.059144] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3916.218110] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3916.588230] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3919.511169] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3920.465183] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 00:08:28 (1717560508) [ 3920.594538] Lustre: Failing over lustre-MDT0000 [ 3922.123166] Lustre: server umount lustre-MDT0000 complete [ 3922.225070] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3922.225315] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3923.005426] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3925.035546] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3925.954705] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 00:08:34 (1717560514) [ 3926.162504] LustreError: 31782:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only *** [ 3926.805920] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3926.818861] Lustre: Failing over lustre-OST0000 [ 3927.241701] Lustre: server umount lustre-OST0000 complete [ 3927.243750] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3927.247365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3928.615151] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3932.267271] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3933.620020] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3937.273963] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3938.628208] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3940.067689] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3940.437867] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3943.284161] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3943.848140] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3947.405411] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3948.658593] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 00:08:56 (1717560536) [ 3948.950425] LustreError: 1590:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only *** [ 3950.636640] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3950.646772] Lustre: Failing over lustre-OST0000 [ 3953.655913] Lustre: server umount lustre-OST0000 complete [ 3953.833663] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3953.839982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3958.659553] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3958.841932] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3963.202311] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3963.219350] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3963.676219] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3965.366904] LustreError: 0-0: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.45@tcp inode [0x20002da91:0x5:0x0] object 0x240000400:14851 extent [0-1048575]: client csum 91eefa91, server csum 449743cf [ 3965.718490] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3968.370099] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3969.103547] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 00:09:17 (1717560557) [ 3969.329983] LustreError: 3764:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only *** [ 3970.055701] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3970.281732] LustreError: 4024:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3971.801997] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3971.917616] Lustre: Failing over lustre-MDT0000 [ 3973.027869] Lustre: server umount lustre-MDT0000 complete [ 3973.029856] LustreError: 11463:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717560563 with bad export cookie 13485144944269890283 [ 3983.028681] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3983.050326] Lustre: Failing over lustre-OST0000 [ 3983.702770] Lustre: server umount lustre-OST0000 complete [ 3988.153588] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3988.157788] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560562/real 1717560562] req@ffff880092468000 x1800988653035456/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560578 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3988.707650] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3989.707626] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3993.161564] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3994.036522] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560567/real 1717560567] req@ffff880092469880 x1800988653035776/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560583 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3994.739618] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560568/real 1717560568] req@ffff880092469f80 x1800988653035840/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560584 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3998.166701] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3998.260351] LustreError: 3210:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8800882e5c00 x1800988653036160/t0(0) o250->MGC192.168.203.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 3999.000046] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3999.046495] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3999.739614] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560573/real 1717560573] req@ffff88009246ad80 x1800988653035968/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560589 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 3999.739629] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4003.289746] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4004.289532] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4004.755486] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560578/real 1717560578] req@ffff880092468e00 x1800988653036032/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560594 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4008.297818] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4009.763479] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4011.846884] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4011.896954] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4016.251181] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4025.237845] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 00:10:05 (1717560605) [ 4025.254460] Lustre: Failing over lustre-OST0000 [ 4026.417189] Lustre: server umount lustre-OST0000 complete [ 4026.561965] Lustre: Failing over lustre-MDT0000 [ 4029.802002] Lustre: server umount lustre-MDT0000 complete [ 4034.803559] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4038.212626] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4038.304063] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4039.035423] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4039.541362] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4039.689096] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4039.689099] LustreError: 6987:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.45@tcp arrived at 1717560630 with bad export cookie 13485144944269894763 [ 4039.689110] LustreError: 6986:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.45@tcp arrived at 1717560630 with bad export cookie 13485144944269894763 [ 4040.688203] LustreError: 17751:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.45@tcp arrived at 1717560630 with bad export cookie 13485144944269894763 [ 4041.931020] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4041.949550] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4042.482102] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4044.089567] Lustre: lustre-OST0000: Denying connection for new client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 4047.491973] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560618/real 1717560618] req@ffff880084c40380 x1800988653044864/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560634 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4049.329552] Lustre: lustre-OST0000: Denying connection for new client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 4052.499687] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560623/real 1717560623] req@ffff880084c42a00 x1800988653044992/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560639 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4053.721649] Lustre: lustre-OST0000: Denying connection for new client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [ 4057.507751] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560628/real 1717560628] req@ffff880084c43480 x1800988653045120/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560644 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4062.515773] Lustre: lustre-OST0000: Denying connection for new client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 4067.523699] Lustre: lustre-OST0000: Denying connection for new client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:48 [ 4072.531666] Lustre: lustre-OST0000: Denying connection for new client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:43 [ 4077.540200] Lustre: lustre-OST0000: Denying connection for new client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:38 [ 4082.547843] Lustre: lustre-OST0000: Denying connection for new client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:33 [ 4087.555762] Lustre: lustre-OST0000: Denying connection for new client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:28 [ 4092.563964] Lustre: lustre-OST0000: Denying connection for new client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:23 [ 4097.571966] Lustre: lustre-OST0000: Denying connection for new client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:18 [ 4102.579713] Lustre: lustre-OST0000: Denying connection for new client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:13 [ 4107.587989] Lustre: lustre-OST0000: Denying connection for new client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:08 [ 4111.426544] Lustre: lustre-OST0000: Denying connection for new client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:03 [ 4111.435534] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 4111.438834] Lustre: 10381:0:(/home/green/git/lustre-release/lustre/obdclass/genops.c:1533:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 9a2607b0-09ea-4a01-aa5e-ce801bdd8dde@ [ 4113.378745] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 4132.979843] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 68 sec [ 4134.631809] Lustre: DEBUG MARKER: free_before: 7517184 free_after: 7517184 [ 4135.639719] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 00:12:04 (1717560724) [ 4135.658388] Lustre: Failing over lustre-OST0000 [ 4136.505819] Lustre: server umount lustre-OST0000 complete [ 4136.507009] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4136.508833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4137.640691] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4141.865830] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4142.643765] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4146.873848] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4147.651242] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4148.493797] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4151.616978] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4152.589644] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 00:12:20 (1717560740) [ 4152.608332] Lustre: Failing over lustre-OST0000 [ 4152.662290] Lustre: server umount lustre-OST0000 complete [ 4154.329780] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4154.333048] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4154.334962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4157.385684] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4157.667684] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4162.393695] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4162.675406] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4165.082830] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4165.513450] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4165.835810] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4181.695203] LustreError: 15176:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 sleeping for 40000ms [ 4181.834488] Lustre: lustre-OST0000: Client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 4181.837612] Lustre: 3210:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560756/real 1717560756] req@ffff88012f3f4e00 x1800988653064384/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1717560772 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_l [ 4197.707345] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 2 clients in recovery for 0:53 [ 4197.844567] Lustre: lustre-OST0000: Client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp) reconnected, waiting for 2 clients in recovery for 0:37 [ 4197.857584] Lustre: 3210:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560772/real 1717560772] req@ffff88012f3f4e00 x1800988653065920/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1717560788 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ptlrpc [ 4205.837582] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 2 clients in recovery for 0:37 [ 4206.355953] LustreError: 15176:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 awake [ 4206.716328] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4209.326178] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4210.461585] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 00:13:18 (1717560798) [ 4210.567112] Lustre: Failing over lustre-MDT0000 [ 4222.537227] Lustre: server umount lustre-MDT0000 complete [ 4222.655570] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4222.665956] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4222.790013] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4222.803667] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4223.553109] LustreError: 17298:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 sleeping for 80000ms [ 4228.689569] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4228.689587] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560803/real 1717560803] req@ffff880087246300 x1800988653069504/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560819 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4233.697596] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560803/real 1717560803] req@ffff880087246d80 x1800988653069568/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560819 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4233.697604] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560808/real 1717560808] req@ffff88008d422a00 x1800988653069760/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560824 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4238.811210] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560808/real 1717560808] req@ffff88008d422680 x1800988653069696/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560824 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4254.841778] Lustre: lustre-MDT0000: Client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp) reconnected, waiting for 1 clients in recovery for 0:53 [ 4270.863879] Lustre: lustre-MDT0000: Client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp) reconnected, waiting for 1 clients in recovery for 0:37 [ 4286.884089] Lustre: lustre-MDT0000: Client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp) reconnected, waiting for 1 clients in recovery for 0:21 [ 4302.812485] Lustre: lustre-MDT0000: Client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp) reconnected, waiting for 1 clients in recovery for 0:05 [ 4302.821832] LustreError: 17298:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 awake [ 4302.826354] LustreError: dumping log to /tmp/lustre-log.1717560893.17298 [ 4303.563309] Lustre: 17298:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2874:target_recovery_thread()) too long recovery - read logs [ 4304.138963] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4307.319683] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4307.834220] Lustre: DEBUG MARKER: == replay-single test 100a: DNE: create striped dir, drop update rep from MDT1, fail MDT1 ========================================================== 00:14:56 (1717560896) [ 4308.435471] Lustre: DEBUG MARKER: SKIP: replay-single test_100a needs >= 2 MDTs [ 4308.954705] Lustre: DEBUG MARKER: == replay-single test 100b: DNE: create striped dir, fail MDT0 ========================================================== 00:14:57 (1717560897) [ 4309.564403] Lustre: DEBUG MARKER: SKIP: replay-single test_100b needs >= 2 MDTs [ 4310.100404] Lustre: DEBUG MARKER: == replay-single test 100c: DNE: create striped dir, abort_recov_mdt mds2 ========================================================== 00:14:58 (1717560898) [ 4310.679422] Lustre: DEBUG MARKER: SKIP: replay-single test_100c needs >= 2 MDTs [ 4311.211144] Lustre: DEBUG MARKER: == replay-single test 100d: DNE: cancel update logs upon recovery abort ========================================================== 00:14:59 (1717560899) [ 4311.814467] Lustre: DEBUG MARKER: SKIP: replay-single test_100d needs > 1 MDTs [ 4312.231553] Lustre: DEBUG MARKER: == replay-single test 100e: DNE: create striped dir on MDT0 and MDT1, fail MDT0, MDT1 ========================================================== 00:15:01 (1717560901) [ 4312.813392] Lustre: DEBUG MARKER: SKIP: replay-single test_100e needs >= 2 MDTs [ 4314.009242] Lustre: DEBUG MARKER: == replay-single test 101: Shouldn't reassign precreated objs to other files after recovery ========================================================== 00:15:02 (1717560902) [ 4314.351413] LustreError: 19258:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 4320.693677] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4320.848482] Lustre: Failing over lustre-MDT0000 [ 4323.231532] Lustre: server umount lustre-MDT0000 complete [ 4323.437281] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4323.443175] Lustre: lustre-MDT0000: Aborting client recovery [ 4323.445719] LustreError: 20619:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2927:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4323.452854] Lustre: 20751:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4323.460069] Lustre: 20751:0:(/home/green/git/lustre-release/lustre/obdclass/genops.c:1533:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 84f7784b-89ad-49c4-870c-988ff78ad8d7@ [ 4323.468845] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 4323.471629] Lustre: 20751:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4324.399244] Lustre: 20751:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4328.393890] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4328.393906] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4339.409564] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4339.409603] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560913/real 1717560913] req@ffff880130f5b800 x1800988653083968/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560929 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4348.068312] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560913/real 1717560913] req@ffff880130f5bb80 x1800988653083904/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560929 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4364.559187] Lustre: DEBUG MARKER: == replay-single test 102a: check resend (request lost) with multiple modify RPCs in flight ========================================================== 00:15:37 (1717560937) [ 4366.852955] Lustre: lustre-MDT0000: Client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp) reconnecting [ 4383.332233] Lustre: DEBUG MARKER: == replay-single test 102b: check resend (reply lost) with multiple modify RPCs in flight ========================================================== 00:15:56 (1717560956) [ 4383.345729] Lustre: lustre-MDT0000: Client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp) reconnecting [ 4383.345743] Lustre: 20630:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff880087349f80 x1800988681866368/t416611831799(0) o36->84f7784b-89ad-49c4-870c-988ff78ad8d7@192.168.203.45@tcp:159/0 lens 488/3152 e 0 to 0 dl 1717560984 ref 1 fl Interpret:/202/0 rc 0/0 job:'chmod.0' uid:0 gid:0 [ 4383.345793] Lustre: 21364:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff880087348380 x1800988681866688/t416611831800(0) o36->84f7784b-89ad-49c4-870c-988ff78ad8d7@192.168.203.45@tcp:159/0 lens 488/3152 e 0 to 0 dl 1717560984 ref 1 fl Interpret:/202/0 rc 0/0 job:'chmod.0' uid:0 gid:0 [ 4383.345847] Lustre: 22848:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff880087348a80 x1800988681867008/t416611831801(0) o36->84f7784b-89ad-49c4-870c-988ff78ad8d7@192.168.203.45@tcp:159/0 lens 488/3152 e 0 to 0 dl 1717560984 ref 1 fl Interpret:/202/0 rc 0/0 job:'chmod.0' uid:0 gid:0 [ 4383.345858] Lustre: 21362:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88008734b800 x1800988681867136/t416611831802(0) o36->84f7784b-89ad-49c4-870c-988ff78ad8d7@192.168.203.45@tcp:159/0 lens 488/3152 e 0 to 0 dl 1717560984 ref 1 fl Interpret:/202/0 rc 0/0 job:'chmod.0' uid:0 gid:0 [ 4383.345889] Lustre: 21364:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff880087349880 x1800988681867584/t416611831804(0) o36->84f7784b-89ad-49c4-870c-988ff78ad8d7@192.168.203.45@tcp:159/0 lens 488/3152 e 0 to 0 dl 1717560984 ref 1 fl Interpret:/202/0 rc 0/0 job:'chmod.0' uid:0 gid:0 [ 4383.413475] Lustre: 22848:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88008734aa00 x1800988681867392/t416611831803(0) o36->84f7784b-89ad-49c4-870c-988ff78ad8d7@192.168.203.45@tcp:159/0 lens 488/3152 e 0 to 0 dl 1717560984 ref 1 fl Interpret:/202/0 rc 0/0 job:'chmod.0' uid:0 gid:0 [ 4385.718109] Lustre: 20630:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88009aacc700 x1800988681866112/t416611831798(0) o36->84f7784b-89ad-49c4-870c-988ff78ad8d7@192.168.203.45@tcp:159/0 lens 488/3152 e 0 to 0 dl 1717560984 ref 1 fl Interpret:/202/0 rc 0/0 job:'chmod.0' uid:0 gid:0 [ 4386.958312] Lustre: DEBUG MARKER: == replay-single test 102c: check replay w/o reconstruction with multiple mod RPCs in flight ========================================================== 00:16:14 (1717560974) [ 4387.298761] LustreError: 24048:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 4389.665741] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4389.781189] Lustre: Failing over lustre-MDT0000 [ 4402.101358] Lustre: server umount lustre-MDT0000 complete [ 4402.230135] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4402.230634] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4403.376677] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4403.383339] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4406.208154] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4406.784890] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4409.404635] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4409.404643] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560983/real 1717560983] req@ffff88012f3f5f80 x1800988653289472/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560999 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4410.466354] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560983/real 1717560983] req@ffff88012f3f4700 x1800988653289408/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717560999 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4413.200609] Lustre: DEBUG MARKER: == replay-single test 102d: check replay [ 4413.329409] Lustre: Failing over lustre-MDT0000 [ 4414.224597] Lustre: server umount lustre-MDT0000 complete [ 4414.227312] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560988/real 1717560988] req@ffff88012f3f5c00 x1800988653289600/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561004 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4425.801939] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717560988/real 1717560988] req@ffff88012f3f4000 x1800988653289664/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561004 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4425.947738] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4425.947741] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4426.877567] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4426.913795] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4426.919762] Lustre: 27271:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88008d48f100 x1800988681890752/t420906795055(0) o36->84f7784b-89ad-49c4-870c-988ff78ad8d7@192.168.203.45@tcp:203/0 lens 488/3152 e 0 to 0 dl 1717561028 ref 1 fl Interpret:/202/0 rc 0/0 job:'chmod.0' uid:0 gid:0 [ 4426.927788] Lustre: 27271:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88008d48f480 x1800988681891392/t420906795057(0) o36->84f7784b-89ad-49c4-870c-988ff78ad8d7@192.168.203.45@tcp:203/0 lens 488/3152 e 0 to 0 dl 1717561028 ref 1 fl Interpret:/202/0 rc 0/0 job:'chmod.0' uid:0 gid:0 [ 4426.935593] Lustre: 27271:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88008d48f800 x1800988681891008/t420906795056(0) o36->84f7784b-89ad-49c4-870c-988ff78ad8d7@192.168.203.45@tcp:203/0 lens 488/3152 e 0 to 0 dl 1717561028 ref 1 fl Interpret:/202/0 rc 0/0 job:'chmod.0' uid:0 gid:0 [ 4426.938158] Lustre: 27271:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88008d48ea00 x1800988681891520/t420906795058(0) o36->84f7784b-89ad-49c4-870c-988ff78ad8d7@192.168.203.45@tcp:203/0 lens 488/3152 e 0 to 0 dl 1717561028 ref 1 fl Interpret:/202/0 rc 0/0 job:'chmod.0' uid:0 gid:0 [ 4426.938303] Lustre: 27585:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88008d48df80 x1800988681891968/t420906795060(0) o36->84f7784b-89ad-49c4-870c-988ff78ad8d7@192.168.203.45@tcp:203/0 lens 488/3152 e 0 to 0 dl 1717561028 ref 1 fl Interpret:/202/0 rc 0/0 job:'chmod.0' uid:0 gid:0 [ 4426.953330] Lustre: 27585:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88008d48fb80 x1800988681891776/t420906795059(0) o36->84f7784b-89ad-49c4-870c-988ff78ad8d7@192.168.203.45@tcp:203/0 lens 488/3152 e 0 to 0 dl 1717561028 ref 1 fl Interpret:/202/0 rc 0/0 job:'chmod.0' uid:0 gid:0 [ 4427.035230] Lustre: 27273:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800882d7100 x1800988681890496/t420906795054(0) o36->84f7784b-89ad-49c4-870c-988ff78ad8d7@192.168.203.45@tcp:203/0 lens 488/3152 e 0 to 0 dl 1717561028 ref 1 fl Interpret:/202/0 rc 0/0 job:'chmod.0' uid:0 gid:0 [ 4429.770424] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4430.359488] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4432.910567] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4432.914844] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561007/real 1717561007] req@ffff880094463480 x1800988653296192/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561023 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4433.965863] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561007/real 1717561007] req@ffff880094461180 x1800988653296256/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561023 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4435.376809] Lustre: DEBUG MARKER: == replay-single test 103: Check otr_next_id overflow ==== 00:17:03 (1717561023) [ 4435.525784] Lustre: Failing over lustre-MDT0000 [ 4437.977580] Lustre: server umount lustre-MDT0000 complete [ 4438.397563] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561012/real 1717561012] req@ffff880094461c00 x1800988653296384/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561028 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4447.758079] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561012/real 1717561012] req@ffff880094462300 x1800988653296448/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561028 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4447.915271] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4447.920493] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.052676] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4451.049061] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4451.883164] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4451.968562] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4451.974922] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561026/real 1717561026] req@ffff88008afd4000 x1800988653302784/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561042 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4452.429290] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561026/real 1717561026] req@ffff88008afd4a80 x1800988653302848/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561042 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4455.864652] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4456.381926] Lustre: DEBUG MARKER: == replay-single test 110a: DNE: create striped dir, fail MDT1 ========================================================== 00:17:25 (1717561045) [ 4456.921558] Lustre: DEBUG MARKER: SKIP: replay-single test_110a needs >= 2 MDTs [ 4456.937550] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561031/real 1717561031] req@ffff88008afd7b80 x1800988653303040/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561047 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4457.006395] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561031/real 1717561031] req@ffff88008afd7480 x1800988653302976/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561047 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4457.527207] Lustre: DEBUG MARKER: == replay-single test 110b: DNE: create striped dir, fail MDT1 and client ========================================================== 00:17:26 (1717561046) [ 4458.141432] Lustre: DEBUG MARKER: SKIP: replay-single test_110b needs >= 2 MDTs [ 4458.619747] Lustre: DEBUG MARKER: == replay-single test 110c: DNE: create striped dir, fail MDT2 ========================================================== 00:17:27 (1717561047) [ 4459.225136] Lustre: DEBUG MARKER: SKIP: replay-single test_110c needs >= 2 MDTs [ 4459.737001] Lustre: DEBUG MARKER: == replay-single test 110d: DNE: create striped dir, fail MDT2 and client ========================================================== 00:17:28 (1717561048) [ 4460.348462] Lustre: DEBUG MARKER: SKIP: replay-single test_110d needs >= 2 MDTs [ 4460.861937] Lustre: DEBUG MARKER: == replay-single test 110e: DNE: create striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 00:17:29 (1717561049) [ 4461.455118] Lustre: DEBUG MARKER: SKIP: replay-single test_110e needs >= 2 MDTs [ 4461.961572] Lustre: DEBUG MARKER: SKIP: replay-single test_110f skipping excluded test 110f [ 4461.965410] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561036/real 1717561036] req@ffff88008afd7100 x1800988653303168/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561052 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4462.039545] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561036/real 1717561036] req@ffff88008afd4e00 x1800988653303232/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561052 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4462.572150] Lustre: DEBUG MARKER: == replay-single test 110g: DNE: create striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 00:17:31 (1717561051) [ 4463.100107] Lustre: DEBUG MARKER: SKIP: replay-single test_110g needs >= 2 MDTs [ 4463.619260] Lustre: DEBUG MARKER: == replay-single test 111a: DNE: unlink striped dir, fail MDT1 ========================================================== 00:17:32 (1717561052) [ 4464.232005] Lustre: DEBUG MARKER: SKIP: replay-single test_111a needs >= 2 MDTs [ 4464.748768] Lustre: DEBUG MARKER: == replay-single test 111b: DNE: unlink striped dir, fail MDT2 ========================================================== 00:17:33 (1717561053) [ 4465.245514] Lustre: DEBUG MARKER: SKIP: replay-single test_111b needs >= 2 MDTs [ 4465.767571] Lustre: DEBUG MARKER: == replay-single test 111c: DNE: unlink striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 00:17:34 (1717561054) [ 4466.388213] Lustre: DEBUG MARKER: SKIP: replay-single test_111c needs >= 2 MDTs [ 4466.908377] Lustre: DEBUG MARKER: == replay-single test 111d: DNE: unlink striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 00:17:35 (1717561055) [ 4467.518533] Lustre: DEBUG MARKER: SKIP: replay-single test_111d needs >= 2 MDTs [ 4468.040103] Lustre: DEBUG MARKER: == replay-single test 111e: DNE: unlink striped dir, uncommit on MDT2, fail MDT1/MDT2 ========================================================== 00:17:36 (1717561056) [ 4468.645188] Lustre: DEBUG MARKER: SKIP: replay-single test_111e needs >= 2 MDTs [ 4469.160703] Lustre: DEBUG MARKER: == replay-single test 111f: DNE: unlink striped dir, uncommit on MDT1, fail MDT1/MDT2 ========================================================== 00:17:37 (1717561057) [ 4469.762573] Lustre: DEBUG MARKER: SKIP: replay-single test_111f needs >= 2 MDTs [ 4470.282046] Lustre: DEBUG MARKER: == replay-single test 111g: DNE: unlink striped dir, fail MDT1/MDT2 ========================================================== 00:17:39 (1717561059) [ 4470.910610] Lustre: DEBUG MARKER: SKIP: replay-single test_111g needs >= 2 MDTs [ 4471.455396] Lustre: DEBUG MARKER: == replay-single test 112a: DNE: cross MDT rename, fail MDT1 ========================================================== 00:17:40 (1717561060) [ 4472.025088] Lustre: DEBUG MARKER: SKIP: replay-single test_112a needs >= 4 MDTs [ 4472.547185] Lustre: DEBUG MARKER: == replay-single test 112b: DNE: cross MDT rename, fail MDT2 ========================================================== 00:17:41 (1717561061) [ 4473.176692] Lustre: DEBUG MARKER: SKIP: replay-single test_112b needs >= 4 MDTs [ 4473.714519] Lustre: DEBUG MARKER: == replay-single test 112c: DNE: cross MDT rename, fail MDT3 ========================================================== 00:17:42 (1717561062) [ 4474.337576] Lustre: DEBUG MARKER: SKIP: replay-single test_112c needs >= 4 MDTs [ 4474.884313] Lustre: DEBUG MARKER: == replay-single test 112d: DNE: cross MDT rename, fail MDT4 ========================================================== 00:17:43 (1717561063) [ 4475.517244] Lustre: DEBUG MARKER: SKIP: replay-single test_112d needs >= 4 MDTs [ 4476.058743] Lustre: DEBUG MARKER: == replay-single test 112e: DNE: cross MDT rename, fail MDT1 and MDT2 ========================================================== 00:17:44 (1717561064) [ 4476.668836] Lustre: DEBUG MARKER: SKIP: replay-single test_112e needs >= 4 MDTs [ 4477.198200] Lustre: DEBUG MARKER: == replay-single test 112f: DNE: cross MDT rename, fail MDT1 and MDT3 ========================================================== 00:17:45 (1717561065) [ 4477.852214] Lustre: DEBUG MARKER: SKIP: replay-single test_112f needs >= 4 MDTs [ 4478.393624] Lustre: DEBUG MARKER: == replay-single test 112g: DNE: cross MDT rename, fail MDT1 and MDT4 ========================================================== 00:17:47 (1717561067) [ 4479.021932] Lustre: DEBUG MARKER: SKIP: replay-single test_112g needs >= 4 MDTs [ 4479.551529] Lustre: DEBUG MARKER: == replay-single test 112h: DNE: cross MDT rename, fail MDT2 and MDT3 ========================================================== 00:17:48 (1717561068) [ 4480.178673] Lustre: DEBUG MARKER: SKIP: replay-single test_112h needs >= 4 MDTs [ 4480.712793] Lustre: DEBUG MARKER: == replay-single test 112i: DNE: cross MDT rename, fail MDT2 and MDT4 ========================================================== 00:17:49 (1717561069) [ 4481.265227] Lustre: DEBUG MARKER: SKIP: replay-single test_112i needs >= 4 MDTs [ 4481.802178] Lustre: DEBUG MARKER: == replay-single test 112j: DNE: cross MDT rename, fail MDT3 and MDT4 ========================================================== 00:17:50 (1717561070) [ 4482.425415] Lustre: DEBUG MARKER: SKIP: replay-single test_112j needs >= 4 MDTs [ 4482.952894] Lustre: DEBUG MARKER: == replay-single test 112k: DNE: cross MDT rename, fail MDT1,MDT2,MDT3 ========================================================== 00:17:51 (1717561071) [ 4483.566976] Lustre: DEBUG MARKER: SKIP: replay-single test_112k needs >= 4 MDTs [ 4484.106275] Lustre: DEBUG MARKER: == replay-single test 112l: DNE: cross MDT rename, fail MDT1,MDT2,MDT4 ========================================================== 00:17:52 (1717561072) [ 4484.721158] Lustre: DEBUG MARKER: SKIP: replay-single test_112l needs >= 4 MDTs [ 4485.245377] Lustre: DEBUG MARKER: == replay-single test 112m: DNE: cross MDT rename, fail MDT1,MDT3,MDT4 ========================================================== 00:17:53 (1717561073) [ 4485.858368] Lustre: DEBUG MARKER: SKIP: replay-single test_112m needs >= 4 MDTs [ 4486.375945] Lustre: DEBUG MARKER: == replay-single test 112n: DNE: cross MDT rename, fail MDT2,MDT3,MDT4 ========================================================== 00:17:55 (1717561075) [ 4486.998198] Lustre: DEBUG MARKER: SKIP: replay-single test_112n needs >= 4 MDTs [ 4487.536507] Lustre: DEBUG MARKER: == replay-single test 115: failover for create/unlink striped directory ========================================================== 00:17:56 (1717561076) [ 4488.153849] Lustre: DEBUG MARKER: SKIP: replay-single test_115 needs >= 2 MDTs [ 4488.685558] Lustre: DEBUG MARKER: == replay-single test 116a: large update log master MDT recovery ========================================================== 00:17:57 (1717561077) [ 4489.315856] Lustre: DEBUG MARKER: SKIP: replay-single test_116a needs >= 2 MDTs [ 4489.850815] Lustre: DEBUG MARKER: == replay-single test 116b: large update log slave MDT recovery ========================================================== 00:17:58 (1717561078) [ 4490.359665] Lustre: DEBUG MARKER: SKIP: replay-single test_116b needs >= 2 MDTs [ 4490.895896] Lustre: DEBUG MARKER: == replay-single test 117: DNE: cross MDT unlink, fail MDT1 and MDT2 ========================================================== 00:17:59 (1717561079) [ 4491.482175] Lustre: DEBUG MARKER: SKIP: replay-single test_117 needs >= 4 MDTs [ 4492.006932] Lustre: DEBUG MARKER: == replay-single test 118: invalidate osp update will not cause update log corruption ========================================================== 00:18:00 (1717561080) [ 4492.607710] Lustre: DEBUG MARKER: SKIP: replay-single test_118 needs >= 2 MDTs [ 4493.129822] Lustre: DEBUG MARKER: == replay-single test 119: timeout of normal replay does not cause DNE replay fails ========================================================== 00:18:01 (1717561081) [ 4493.758036] Lustre: DEBUG MARKER: SKIP: replay-single test_119 needs >= 2 MDTs [ 4494.283570] Lustre: DEBUG MARKER: == replay-single test 120: DNE fail abort should stop both normal and DNE replay ========================================================== 00:18:02 (1717561082) [ 4494.880206] Lustre: DEBUG MARKER: SKIP: replay-single test_120 needs >= 2 MDTs [ 4495.949854] Lustre: DEBUG MARKER: == replay-single test 121: lock replay timed out and race ========================================================== 00:18:04 (1717561084) [ 4496.084753] Lustre: Failing over lustre-MDT0000 [ 4499.701466] Lustre: server umount lustre-MDT0000 complete [ 4499.849053] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4499.849056] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4500.885046] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4501.997596] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4513.897565] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4513.897570] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561088/real 1717561088] req@ffff88013034f100 x1800988653315264/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561104 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4518.014048] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561088/real 1717561088] req@ffff88013034ea00 x1800988653315200/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561104 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4518.032465] Lustre: lustre-MDT0000: Client 84f7784b-89ad-49c4-870c-988ff78ad8d7 (at 192.168.203.45@tcp) reconnected, waiting for 1 clients in recovery for 0:53 [ 4518.038079] Lustre: 3626:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/tgt_handler.c:715:process_req_last_xid()) @@@ unexpected xid=665fd5aa15fc0 != exp_last_xid=665fd5aa1603f, rc = -71 req@ffff880088e73800 x1800988681920448/t0(0) o101->84f7784b-89ad-49c4-870c-988ff78ad8d7@192.168.203.45@tcp:0/0 lens 328/0 e 0 to 0 dl 1717561103 ref 1 fl Interpret:/240/ffffffff rc [ 4522.161302] Lustre: 3626:0:(/home/green/git/lustre-release/lustre/ptlrpc/service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (11/5s); client may timeout req@ffff880088e73800 x1800988681920448/t0(0) o101->84f7784b-89ad-49c4-870c-988ff78ad8d7@192.168.203.45@tcp:0/0 lens 328/224 e 0 to 0 dl 1717561103 ref 1 fl Complete:/240/0 rc -71/-71 job: [ 4523.278423] Lustre: DEBUG MARKER: == replay-single test 130a: DoM file create (setstripe) replay ========================================================== 00:18:31 (1717561111) [ 4523.608183] LustreError: 4897:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 4524.327636] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4524.466129] Lustre: Failing over lustre-MDT0000 [ 4536.768798] Lustre: server umount lustre-MDT0000 complete [ 4536.905888] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4536.905932] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4537.059753] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4538.041897] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4540.274507] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4540.635082] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4540.922587] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4540.956575] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561115/real 1717561115] req@ffff88008afd4e00 x1800988653321664/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561131 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4544.024877] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561115/real 1717561115] req@ffff88008afd6300 x1800988653321728/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561131 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4544.941553] Lustre: DEBUG MARKER: == replay-single test 130b: DoM file create (inherited) replay ========================================================== 00:18:53 (1717561133) [ 4545.219944] LustreError: 7033:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 4545.843037] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4545.939869] Lustre: Failing over lustre-MDT0000 [ 4546.842503] Lustre: server umount lustre-MDT0000 complete [ 4546.844331] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561120/real 1717561120] req@ffff88008afd7800 x1800988653321856/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561136 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4550.850504] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561120/real 1717561120] req@ffff88008afd6d80 x1800988653321920/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561136 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4550.953542] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561125/real 1717561125] req@ffff88008afd4a80 x1800988653322112/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561141 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4558.078099] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561125/real 1717561125] req@ffff88008afd4380 x1800988653322048/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561141 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4558.221696] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4558.229852] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4559.281205] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4561.311014] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4562.193267] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4562.761035] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4563.263567] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4563.268091] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561137/real 1717561137] req@ffff88008bc13480 x1800988653327104/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561153 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4566.419703] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561137/real 1717561137] req@ffff88008bc12a00 x1800988653327040/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561153 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4567.537604] Lustre: DEBUG MARKER: == replay-single test 131a: DoM file write lock replay === 00:19:15 (1717561155) [ 4567.790287] LustreError: 9299:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 4568.225500] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4568.257544] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561142/real 1717561142] req@ffff88008bc11880 x1800988653327296/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561158 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4568.505753] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561142/real 1717561142] req@ffff88008bc11500 x1800988653327232/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561158 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4568.641601] Lustre: Failing over lustre-MDT0000 [ 4573.505575] Lustre: server umount lustre-MDT0000 complete [ 4574.273577] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561147/real 1717561147] req@ffff88008737b800 x1800988653327424/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561163 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4580.611348] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561147/real 1717561147] req@ffff88008737b480 x1800988653327488/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561163 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4580.705323] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4580.705492] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4581.773309] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.147621] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4584.646421] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4585.199703] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4588.733566] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4588.738497] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561163/real 1717561163] req@ffff880094461f80 x1800988653332800/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561179 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4588.803706] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561163/real 1717561163] req@ffff880094460700 x1800988653332864/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561179 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4589.403513] Lustre: DEBUG MARKER: SKIP: replay-single test_131b skipping excluded test 131b [ 4590.492957] Lustre: DEBUG MARKER: == replay-single test 132a: PFL new component instantiate replay ========================================================== 00:19:38 (1717561178) [ 4590.825181] LustreError: 11591:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 4591.640745] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4591.764254] Lustre: Failing over lustre-MDT0000 [ 4593.638606] Lustre: server umount lustre-MDT0000 complete [ 4593.641304] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561168/real 1717561168] req@ffff880094463b80 x1800988653333056/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561184 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4604.053884] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561168/real 1717561168] req@ffff880094461880 x1800988653332992/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561184 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4604.206627] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4604.206696] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4605.329786] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4607.187160] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4608.167598] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4608.687273] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4612.222383] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4612.225563] Lustre: DEBUG MARKER: == replay-single test 133: check resend of ongoing requests for lwp during failover ========================================================== 00:20:01 (1717561201) [ 4612.226557] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561186/real 1717561186] req@ffff880087378380 x1800988653338560/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561202 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4612.762305] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561186/real 1717561186] req@ffff88008737b480 x1800988653338624/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561202 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4613.365650] Lustre: DEBUG MARKER: SKIP: replay-single test_133 needs >= 2 MDTs [ 4617.436565] Lustre: DEBUG MARKER: == replay-single test 134: replay creation of a file created in a pool ========================================================== 00:20:02 (1717561202) [ 4617.445957] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561191/real 1717561191] req@ffff880087379c00 x1800988653338816/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561207 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4619.948839] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561191/real 1717561191] req@ffff88008737a300 x1800988653338752/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561207 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4620.281109] LustreError: 14265:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 4620.941224] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4621.067685] Lustre: Failing over lustre-MDT0000 [ 4633.459708] Lustre: server umount lustre-MDT0000 complete [ 4633.604585] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4633.608702] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4634.727065] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4636.733478] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4637.629128] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4638.152587] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4639.656570] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4639.661467] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561214/real 1717561214] req@ffff88009ac87100 x1800988653350528/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561230 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4644.625622] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561214/real 1717561214] req@ffff88009ac85880 x1800988653350592/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561230 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4644.641510] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561219/real 1717561219] req@ffff88008d421f80 x1800988653350784/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561235 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4646.740971] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561219/real 1717561219] req@ffff88009ac86680 x1800988653350720/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717561235 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 4647.417653] Lustre: DEBUG MARKER: == replay-single test 135: Server failure in lock replay phase ========================================================== 00:20:35 (1717561235) [ 4647.440038] Lustre: Failing over lustre-OST0000 [ 4648.634082] Lustre: server umount lustre-OST0000 complete [ 4648.636664] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4648.641409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4652.244842] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4653.657887] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4657.236261] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4658.665865] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4661.558488] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4661.692590] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4664.359012] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4664.903909] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4668.588490] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4668.910827] LustreError: 18503:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only *** [ 4669.679766] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4669.728375] Lustre: Failing over lustre-OST0000 [ 4669.833853] Lustre: server umount lustre-OST0000 complete [ 4669.837999] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4671.701487] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4672.264907] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 4674.806836] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4675.293907] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4676.801739] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4677.344286] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount REPLAY_LOCKS osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4678.028659] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in REPLAY_LOCKS state after 0 sec [ 4678.034916] Lustre: Failing over lustre-OST0000 [ 4678.037386] LustreError: 20671:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2927:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery [ 4678.045929] Lustre: 19816:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4678.065246] Lustre: 19816:0:(/home/green/git/lustre-release/lustre/ofd/ofd_obd.c:557:ofd_postrecov()) lustre-OST0000: auto trigger paused LFSCK failed: rc = -6 [ 4690.100673] Lustre: server umount lustre-OST0000 complete [ 4691.568579] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 4691.573423] Lustre: 3210:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717561266/real 1717561266] req@ffff88009ac86a00 x1800988653367552/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1717561282 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_l [ 4691.662500] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4693.727965] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4693.772703] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4696.986095] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4701.154528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4701.683893] Lustre: server umount lustre-OST0000 complete [ 4701.994048] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4703.722250] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4703.729414] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4706.691693] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4707.001847] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4708.703795] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4711.552570] Lustre: server umount lustre-OST0001 complete [ 4711.699776] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4712.837026] LustreError: 137-5: lustre-OST0001: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4713.303815] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4717.586974] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4717.723369] LustreError: 167-0: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4722.112165] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4722.631800] Lustre: DEBUG MARKER: == replay-single test 136: MDS to disconnect all OSPs first, then cleanup ldlm ========================================================== 00:21:51 (1717561311) [ 4723.280798] Lustre: DEBUG MARKER: SKIP: replay-single test_136 needs > 2 MDTs [ 4723.816334] Lustre: DEBUG MARKER: == replay-single test 137a: DNE: create under striped dir, fail MDT1 ========================================================== 00:21:52 (1717561312) [ 4724.442801] Lustre: DEBUG MARKER: SKIP: replay-single test_137a needs >= 2 MDTs [ 4724.996479] Lustre: DEBUG MARKER: == replay-single test 137b: DNE: create under striped dir, fail MDT2 ========================================================== 00:21:53 (1717561313) [ 4725.608796] Lustre: DEBUG MARKER: SKIP: replay-single test_137b needs >= 2 MDTs [ 4726.139907] Lustre: DEBUG MARKER: == replay-single test 137c: DNE: create under striped dir, fail MDT1/MDT2 ========================================================== 00:21:54 (1717561314) [ 4726.762111] Lustre: DEBUG MARKER: SKIP: replay-single test_137c needs >= 2 MDTs [ 4727.217074] Lustre: DEBUG MARKER: == replay-single test 200: Dropping one OBD_PING should not cause disconnect ========================================================== 00:21:55 (1717561315) [ 4727.848569] Lustre: DEBUG MARKER: SKIP: replay-single test_200 Need remote client [ 4730.968187] Lustre: DEBUG MARKER: == replay-single test complete, duration 4654 sec ======== 00:21:57 (1717561317) [ 4731.017690] Lustre: Failing over lustre-MDT0000 [ 4731.020018] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4731.124699] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4743.540812] Lustre: server umount lustre-MDT0000 complete [ 4744.830743] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4746.757713] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4747.694496] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4748.268016] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4752.586505] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4754.038234] Lustre: server umount lustre-MDT0000 complete [ 4754.039686] LustreError: 17751:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717561344 with bad export cookie 13485144944270112253 [ 4754.039717] LustreError: 8612:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717561344 with bad export cookie 13485144944270112253 [ 4754.039757] LustreError: 11463:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717561344 with bad export cookie 13485144944270112253 [ 4754.053675] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4755.520589] Lustre: server umount lustre-OST0000 complete [ 4759.708243] Lustre: server umount lustre-OST0001 complete [ 4760.438059] Key type lgssc unregistered [ 4760.525914] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local