[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 319891033 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.456537] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.460575] pid_max: default: 32768 minimum: 301 [ 0.462549] Security Framework initialized [ 0.464360] SELinux: Initializing. [ 0.466955] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.470227] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.472425] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.473928] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.476033] Initializing cgroup subsys memory [ 0.477016] Initializing cgroup subsys devices [ 0.478091] Initializing cgroup subsys freezer [ 0.479074] Initializing cgroup subsys net_cls [ 0.480088] Initializing cgroup subsys blkio [ 0.480958] Initializing cgroup subsys perf_event [ 0.481932] Initializing cgroup subsys hugetlb [ 0.482863] Initializing cgroup subsys pids [ 0.483813] Initializing cgroup subsys net_prio [ 0.485037] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.487615] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.488914] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.490085] tlb_flushall_shift: 6 [ 0.490857] FEATURE SPEC_CTRL Present [ 0.491679] FEATURE IBPB_SUPPORT Present [ 0.492708] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.494199] Spectre V2 : Vulnerable [ 0.495004] Speculative Store Bypass: Vulnerable [ 0.496760] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.502719] ACPI: Core revision 20130517 [ 0.504857] ACPI: All ACPI Tables successfully acquired [ 0.506269] ftrace: allocating 30294 entries in 119 pages [ 0.552229] Enabling x2apic [ 0.552973] Enabled x2apic [ 0.554002] Switched APIC routing to physical x2apic. [ 0.558172] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.560627] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.564147] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.566113] ... version: 2 [ 0.567407] ... bit width: 48 [ 0.568674] ... generic registers: 4 [ 0.570456] ... value mask: 0000ffffffffffff [ 0.572005] ... max period: 00007fffffffffff [ 0.574433] ... fixed-purpose events: 3 [ 0.576187] ... event mask: 000000070000000f [ 0.578628] KVM setup paravirtual spinlock [ 0.581464] smpboot: Booting Node 0, Processors #1[ 0.582861] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.585376] KVM setup async PF for cpu 1 [ 0.586318] kvm-stealtime: cpu 1, msr 13e2935c0 #2[ 0.588007] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.590529] KVM setup async PF for cpu 2 [ 0.591002] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock #3 OK [ 0.592677] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.593741] Brought up 4 CPUs [ 0.593758] KVM setup async PF for cpu 3 [ 0.593764] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.596492] smpboot: Max logical packages: 1 [ 0.597564] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.600909] devtmpfs: initialized [ 0.601772] x86/mm: Memory block size: 128MB [ 0.605553] EVM: security.selinux [ 0.606300] EVM: security.ima [ 0.607054] EVM: security.capability [ 0.609515] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.611345] NET: Registered protocol family 16 [ 0.612638] cpuidle: using governor haltpoll [ 0.613891] ACPI: bus type PCI registered [ 0.614909] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.616530] PCI: Using configuration type 1 for base access [ 0.617890] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.625180] ACPI: Added _OSI(Module Device) [ 0.626675] ACPI: Added _OSI(Processor Device) [ 0.628716] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.630871] ACPI: Added _OSI(Processor Aggregator Device) [ 0.633382] ACPI: Added _OSI(Linux-Dell-Video) [ 0.639232] ACPI: Interpreter enabled [ 0.640754] ACPI: (supports S0 S3 S4 S5) [ 0.642050] ACPI: Using IOAPIC for interrupt routing [ 0.643570] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.646362] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.652782] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.655115] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.657463] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.659519] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.664409] acpiphp: Slot [2] registered [ 0.665910] acpiphp: Slot [3] registered [ 0.667108] acpiphp: Slot [4] registered [ 0.668817] acpiphp: Slot [5] registered [ 0.670620] acpiphp: Slot [6] registered [ 0.672185] acpiphp: Slot [7] registered [ 0.673612] acpiphp: Slot [8] registered [ 0.674917] acpiphp: Slot [9] registered [ 0.676145] acpiphp: Slot [10] registered [ 0.677501] acpiphp: Slot [11] registered [ 0.678717] acpiphp: Slot [12] registered [ 0.679991] acpiphp: Slot [13] registered [ 0.681265] acpiphp: Slot [14] registered [ 0.682529] acpiphp: Slot [15] registered [ 0.684377] acpiphp: Slot [16] registered [ 0.685966] acpiphp: Slot [17] registered [ 0.687240] acpiphp: Slot [18] registered [ 0.688549] acpiphp: Slot [19] registered [ 0.689994] acpiphp: Slot [20] registered [ 0.691149] acpiphp: Slot [21] registered [ 0.692466] acpiphp: Slot [22] registered [ 0.693628] acpiphp: Slot [23] registered [ 0.694946] acpiphp: Slot [24] registered [ 0.696218] acpiphp: Slot [25] registered [ 0.697580] acpiphp: Slot [26] registered [ 0.699449] acpiphp: Slot [27] registered [ 0.701627] acpiphp: Slot [28] registered [ 0.703267] acpiphp: Slot [29] registered [ 0.704677] acpiphp: Slot [30] registered [ 0.706374] acpiphp: Slot [31] registered [ 0.707782] PCI host bridge to bus 0000:00 [ 0.709108] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.711153] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.713595] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.715906] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.718096] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.720391] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.729912] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.731695] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.733148] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.734674] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.737250] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.739400] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.879931] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.882305] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.884893] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.887089] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.889135] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.892652] vgaarb: loaded [ 0.893735] SCSI subsystem initialized [ 0.894991] ACPI: bus type USB registered [ 0.896371] usbcore: registered new interface driver usbfs [ 0.898556] usbcore: registered new interface driver hub [ 0.900672] usbcore: registered new device driver usb [ 0.902992] PCI: Using ACPI for IRQ routing [ 0.905697] NetLabel: Initializing [ 0.907177] NetLabel: domain hash size = 128 [ 0.908908] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.910906] NetLabel: unlabeled traffic allowed by default [ 0.913577] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.915701] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.921330] amd_nb: Cannot enumerate AMD northbridges [ 0.923526] Switched to clocksource kvm-clock [ 0.943420] pnp: PnP ACPI init [ 0.944609] ACPI: bus type PNP registered [ 0.947292] pnp: PnP ACPI: found 6 devices [ 0.948817] ACPI: bus type PNP unregistered [ 0.960578] NET: Registered protocol family 2 [ 0.962974] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 0.966357] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 0.970360] TCP: Hash tables configured (established 32768 bind 32768) [ 0.973128] TCP: reno registered [ 0.974545] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 0.977157] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 0.979937] NET: Registered protocol family 1 [ 0.981178] RPC: Registered named UNIX socket transport module. [ 0.982268] RPC: Registered udp transport module. [ 0.983067] RPC: Registered tcp transport module. [ 0.983885] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.985014] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.986121] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.987151] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.988421] Unpacking initramfs... [ 2.331735] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 2.333977] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.335145] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 2.337030] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 2.338486] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 2.339638] RAPL PMU: hw unit of domain package 2^-0 Joules [ 2.340609] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 2.343073] cryptomgr_test (49) used greatest stack depth: 14128 bytes left [ 2.343453] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 2.343499] Initialise system trusted keyring [ 2.375003] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 2.376201] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 2.381310] zpool: loaded [ 2.381864] zbud: loaded [ 2.382654] VFS: Disk quotas dquot_6.6.0 [ 2.383801] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.385550] NFS: Registering the id_resolver key type [ 2.386601] Key type id_resolver registered [ 2.387411] Key type id_legacy registered [ 2.388140] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 2.389676] Key type big_key registered [ 2.391713] cryptomgr_test (55) used greatest stack depth: 13968 bytes left [ 2.394772] cryptomgr_test (57) used greatest stack depth: 13664 bytes left [ 2.396505] cryptomgr_test (60) used greatest stack depth: 13536 bytes left [ 2.399064] NET: Registered protocol family 38 [ 2.400093] Key type asymmetric registered [ 2.401672] Asymmetric key parser 'x509' registered [ 2.402815] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 2.404981] io scheduler noop registered [ 2.406687] io scheduler deadline registered (default) [ 2.408549] io scheduler cfq registered [ 2.409749] io scheduler mq-deadline registered [ 2.410957] io scheduler kyber registered [ 2.414093] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 2.416004] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 2.418681] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.420266] ACPI: Power Button [PWRF] [ 2.421354] GHES: HEST is not enabled! [ 2.476097] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 2.532765] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 2.666213] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 2.722019] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 2.836648] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.863673] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 2.890889] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 2.893384] Non-volatile memory driver v1.3 [ 2.894386] Linux agpgart interface v0.103 [ 2.895628] crash memory driver: version 1.1 [ 2.897035] nbd: registered device at major 43 [ 2.907218] virtio_blk virtio1: [vda] 57680 512-byte logical blocks (29.5 MB/28.1 MiB) [ 2.920479] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.932109] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.943899] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.957658] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.969588] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.979373] rdac: device handler registered [ 2.981170] hp_sw: device handler registered [ 2.982734] emc: device handler registered [ 2.984262] libphy: Fixed MDIO Bus: probed [ 2.991645] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 2.994004] ehci-pci: EHCI PCI platform driver [ 2.995624] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 2.997819] ohci-pci: OHCI PCI platform driver [ 2.999399] uhci_hcd: USB Universal Host Controller Interface driver [ 3.001954] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.006103] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.008278] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.010762] mousedev: PS/2 mouse device common for all mice [ 3.013692] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.014150] rtc_cmos 00:05: RTC can wake from S4 [ 3.015371] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 3.016076] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.019550] hidraw: raw HID events driver (C) Jiri Kosina [ 3.019798] usbcore: registered new interface driver usbhid [ 3.019798] usbhid: USB HID core driver [ 3.019866] drop_monitor: Initializing network drop monitor service [ 3.019914] Netfilter messages via NETLINK v0.30. [ 3.019974] TCP: cubic registered [ 3.019980] Initializing XFRM netlink socket [ 3.020221] NET: Registered protocol family 10 [ 3.020653] NET: Registered protocol family 17 [ 3.020714] Key type dns_resolver registered [ 3.021068] mce: Using 10 MCE banks [ 3.021307] Loading compiled-in X.509 certificates [ 3.022272] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 3.022308] registered taskstats version 1 [ 3.024214] modprobe (70) used greatest stack depth: 13456 bytes left [ 3.025932] Key type trusted registered [ 3.029018] Key type encrypted registered [ 3.029079] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 3.031470] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 3.032259] rtc_cmos 00:05: setting system clock to 2024-06-05 03:03:12 UTC (1717556592) [ 3.058419] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 3.060163] Write protecting the kernel read-only data: 12288k [ 3.061663] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 3.063295] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 3.070800] random: systemd: uninitialized urandom read (16 bytes read) [ 3.073589] random: systemd: uninitialized urandom read (16 bytes read) [ 3.075057] random: systemd: uninitialized urandom read (16 bytes read) [ 3.078593] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 3.083309] systemd[1]: Detected virtualization kvm. [ 3.084386] systemd[1]: Detected architecture x86-64. [ 3.085456] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-572.el7 (Initramfs)! [ 3.088703] systemd[1]: No hostname configured. [ 3.089677] systemd[1]: Set hostname to . [ 3.090858] random: systemd: uninitialized urandom read (16 bytes read) [ 3.092226] systemd[1]: Initializing machine ID from random generator. [ 3.100222] mkdir (87) used greatest stack depth: 13376 bytes left [ 3.102843] ln (88) used greatest stack depth: 13008 bytes left [ 3.133228] random: systemd: uninitialized urandom read (16 bytes read) [ 3.136123] random: systemd: uninitialized urandom read (16 bytes read) [ 3.138969] random: systemd: uninitialized urandom read (16 bytes read) [ 3.141758] random: systemd: uninitialized urandom read (16 bytes read) [ 3.146246] random: systemd: uninitialized urandom read (16 bytes read) [ 3.149320] random: systemd: uninitialized urandom read (16 bytes read) [ 3.161601] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.165475] systemd[1]: Created slice Root Slice. [ OK ] Created slice Root Slice. [ 3.167879] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.171152] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.173990] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.176700] systemd[1]: Reached target Sockets. [ OK ] Reached target Sockets. [ 3.179642] systemd[1]: Created slice System Slice. [ OK ] Created slice System Slice. [ 3.183176] systemd[1]: Starting Journal Service... Starting Journal Service... [ 3.185459] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.188834] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ 3.196252] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ 3.199497] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 3.201970] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.204808] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.210902] systemd[1]: Starting Load Kernel Modules... Starting Load Kernel Modules... [ 3.216365] systemd[1]: Started Create list of required static device nodes for the current kernel. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ 3.224151] systemd[1]: Started Setup Virtual Console. [ OK ] Started Setup Virtual Console. [ 3.227822] systemd[1]: Started Load Kernel Modules. [ OK ] Started Load Kernel Modules. [ 3.232008] systemd[1]: Started Journal Service. [ OK ] Started Journal Service. Starting Apply Kernel Variables... Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ 3.344655] tsc: Refined TSC clocksource calibration: 2399.965 MHz [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.447058] random: fast init done [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Configuration File System... [ OK ] Mounted Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ 3.723899] scsi host0: ata_piix [ 3.729831] scsi host1: ata_piix [ 3.730837] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 3.732718] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 Starting Show Plymouth Boot Screen... Starting dracut initqueue hook... %G[ OK ] Started Show Plymouth Boot Screen. [ OK ] Started Forward Password Requests to Plymouth Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. [ 3.862408] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 3.865918] ip (342) used greatest stack depth: 12464 bytes left [ 3.879807] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 3.992627] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 5.984528] dracut-initqueue[295]: RTNETLINK answers: File exists [ 6.180821] dracut-initqueue[295]: bs=4096, sz=32212254720 bytes [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... Mounting /sysroot... [ 6.919887] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Starting Plymouth switch root service... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Timers. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Load Kernel Modules. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Plymouth switch root service. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 7.348002] systemd-journald[100]: Received SIGTERM from PID 1 (systemd). [ 7.562670] SELinux: Disabled at runtime. [ 7.624143] ip_tables: (C) 2000-2006 Netfilter Core Team [ 7.627419] systemd[1]: Inserted module 'ip_tables' Welcome to CentOS Linux 7 (Core)! [ OK ] Stopped Switch Root. [ OK ] Stopped Journal Service. Starting Journal Service... Mounting Debug File System... [ OK ] Listening on /dev/initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... Starting Load Kernel Modules... Starting Read and set NIS domainname from /etc/sysconfig/network... Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st... nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-selinux\x2dpol...grate\x2dlocal\x2dchanges.slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Delayed Shutdown Socket. Starting Set Up Additional Binary Formats... [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Debug File System. [ OK ] Started Load Kernel Modules. [ OK ] Started Create list of required sta...ce nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Started Read and set NIS domainname from /etc/sysconfig/network. Mounting Arbitrary Executable File Formats File System... [ OK ] Mounted Arbitrary Executable File Formats File System. [ OK ] Started udev Coldplug all Devices. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Mounted /mnt. [ OK ] Started Set Up Additional Binary Formats. [ 8.038154] systemd-journald[564]: Received request to flush runtime journal from PID 1 [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Kernel Device Manager. [ 8.150283] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ OK ] Found device /dev/ttyS1. [ 8.182915] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ OK ] Found device /dev/ttyS0. [ 8.203824] cryptd: max_cpu_qlen set to 1000 [ OK ] Found device /dev/disk/by-label/SWAP. Activating swap /dev/disk/by-label/SWAP... [ OK ] Found device /dev/vda. [ 8.239123] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 8.250102] AVX version of gcm_enc/dec engaged. [ 8.252045] AES CTR mode by8 optimization enabled Mounting /home/green/git/lustre-release... [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. [ 8.298370] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 8.298639] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 8.302743] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ OK ] Mounted /home/green/git/lustre-release. %G[ 8.406899] EDAC MC: Ver: 3.0.0 [ 8.411899] EDAC sbridge: Ver: 1.1.2 [ 9.811715] mount.nfs (768) used greatest stack depth: 10704 bytes left [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Journal Catalog... Starting Preprocess NFS configuration... Starting Tell Plymouth To Write Out Runtime Data... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [FAILED] Failed to start Rebuild Journal Catalog. See 'systemctl status systemd-journal-catalog-update.service' for details. [ OK ] Started Preprocess NFS configuration. [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ OK ] Started Load/Save Random Seed. [FAILED] Failed to start Create Volatile Files and Directories. See 'systemctl status systemd-tmpfiles-setup.service' for details. Starting Update UTMP about System Boot/Shutdown... Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Started Flexible branding. [ OK ] Reached target Paths. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... Starting GSSAPI Proxy Daemon... Starting Dump dmesg to /var/log/dmesg... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Login Service. [ OK ] Started Dump dmesg to /var/log/dmesg. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting /etc/rc.d/rc.local Compatibility... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started /etc/rc.d/rc.local Compatibility. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... Starting Terminate Plymouth Boot Screen... Starting Wait for Plymouth Boot Screen to Quit... [ OK ] Started Network Manager Script Dispatcher Service. CentOS Linux 7 (Core) Kernel 3.10.0-7.9-debug on an x86_64 oleg242-server login: [ 18.569149] device-mapper: uevent: version 1.0.3 [ 18.570569] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 22.604869] libcfs: loading out-of-tree module taints kernel. [ 22.606440] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 22.630238] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 26.977000] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local[ 27.142670] alg: No test for adler32 (adler32-zlib) [ 27.893610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 29.670685] Key type lgssc registered [ 32.495221] icp: module license 'CDDL' taints kernel. [ 32.496783] Disabling lock debugging due to kernel taint [ 35.035227] ZFS: Loaded module v0.8.6-1, ZFS pool version 5000, ZFS filesystem version 5 [ 38.092907] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 42.478005] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 44.502116] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 44.506795] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 46.523311] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 46.527261] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 49.422311] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local[ 52.349822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 52.364563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'[ 52.370030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 53.471646] Lustre: lustre-MDT0000: new disk, initializing[ 53.508329] mount.lustre (7096) used greatest stack depth: 10184 bytes left [ 54.190988] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all[ 58.368407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 58.390767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 58.453577] Lustre: lustre-MDT0001: new disk, initializing [ 59.188230] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck allLDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 63.733275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 63.748136] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 63.751428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 63.809694] Lustre: lustre-OST0000: new disk, initializing[ 63.836213] random: crng init done [ 64.991916] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck allLDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 69.420385] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 69.439174] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 69.442175] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 69.464814] Lustre: lustre-OST0001: new disk, initializing [ 70.713235] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck allLustre: DEBUG MARKER: Using TIMEOUT=20 [ 86.014261] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing check_logdir /tmp/testlogs/Lustre: DEBUG MARKER: oleg242-server.virtnet: executing yml_node [ 87.850253] Lustre: DEBUG MARKER: Client: 2.15.63.95Lustre: DEBUG MARKER: MDS: 2.15.63.95 [ 88.993461] Lustre: DEBUG MARKER: OSS: 2.15.63.95Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-scrub ============----- Tue Jun 4 23:04:37 EDT 2024 [ 92.758082] Lustre: DEBUG MARKER: excepting tests:Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 97.284872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to completeLustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 97.291468] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 101.420258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 102.292871] Lustre: server umount lustre-MDT0000 complete [ 102.292874] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 102.292968] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 102.293007] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 103.553334] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 103.555128] LustreError: 10720:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556693 with bad export cookie 18429265761951025951 [ 103.555139] LustreError: 7105:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556693 with bad export cookie 18429265761951025951 [ 103.555145] LustreError: 7104:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556693 with bad export cookie 18429265761951025951 [ 103.555231] LustreError: 8257:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556693 with bad export cookie 18429265761951025951 [ 103.660528] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 115.948529] Lustre: server umount lustre-MDT0001 complete [ 128.267979] Lustre: server umount lustre-OST0000 complete [ 129.933643] device-mapper: core: cleaned up [ 132.381662] Lustre: server umount lustre-OST0001 complete [ 132.964091] Key type lgssc unregistered [ 133.030916] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 135.090722] device-mapper: uevent: version 1.0.3 [ 135.093299] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 139.152697] LNet: 16807:0:(/home/green/git/lustre-release/lnet/lnet/lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 143.296142] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 143.429543] alg: No test for adler32 (adler32-zlib) [ 144.180646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 145.801727] Key type lgssc registered [ 148.814563] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 150.868203] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 152.668225] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 152.671544] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 152.674431] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 154.472495] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 154.475171] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 157.475654] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local[ 160.477839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 160.492934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'[ 160.497218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 161.592153] Lustre: lustre-MDT0000: new disk, initializing [ 162.335383] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all[ 166.786289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 166.804575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 166.871181] Lustre: lustre-MDT0001: new disk, initializing [ 167.749477] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck allLDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 172.336585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 172.352744] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 172.356260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 172.425843] Lustre: lustre-OST0000: new disk, initializing [ 173.620565] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck allLDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 177.706603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 177.723675] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 177.726553] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 177.747370] Lustre: lustre-OST0001: new disk, initializing [ 178.961681] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck allLustre: DEBUG MARKER: Using TIMEOUT=20 [ 195.402631] Lustre: DEBUG MARKER: == sanity-scrub test 0: Do not auto trigger OI scrub for non-backup/restore case ========================================================== 23:06:23 (1717556783)Lustre: Failing over lustre-MDT0000 [ 201.385984] Lustre: server umount lustre-MDT0000 completeLustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 201.943941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to completeLustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 202.406095] LustreError: 24335:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556791 with bad export cookie 11797782520123723758Lustre: Failing over lustre-MDT0001 [ 202.407040] LustreError: 23131:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556791 with bad export cookie 11797782520123723758 [ 202.407042] LustreError: 21978:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556791 with bad export cookie 11797782520123723758 [ 202.407077] LustreError: 25591:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556791 with bad export cookie 11797782520123723758 [ 202.407089] LustreError: 23131:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556791 with bad export cookie 11797782520123723758 [ 202.531085] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 204.874946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 212.406821] Lustre: server umount lustre-MDT0001 complete [ 212.489132] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8800a6cfc000 x1800988768913088/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 213.238191] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 215.308136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 215.368349] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.368374] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 215.368381] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 215.368448] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 215.381837] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 216.086753] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 217.554364] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.559666] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 221.391643] Lustre: lustre-MDT0000: Denying connection for new client d2680e22-c4ad-40ba-b772-8a5628978505 (at 192.168.202.42@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 221.406600] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556794/real 1717556794] req@ffff8800a6d28380 x1800988768912640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556810 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 221.412574] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556794/real 1717556794] req@ffff8800a6d29500 x1800988768912704/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556810 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 221.427593] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556794/real 1717556794] req@ffff8800a6d2aa00 x1800988768912768/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556810 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 223.396709] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556794/real 1717556794] req@ffff8800a6d29c00 x1800988768912576/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556810 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 223.418790] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556796/real 1717556796] req@ffff8800a6d29f80 x1800988768912896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556812 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 223.431496] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556796/real 1717556796] req@ffff8800a6d2b800 x1800988768912960/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556812 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 223.440559] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556796/real 1717556796] req@ffff8800a6d2bb80 x1800988768913024/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556812 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 224.066087] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556796/real 1717556796] req@ffff8800a6d28e00 x1800988768912832/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556812 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 228.517636] Lustre: DEBUG MARKER: == sanity-scrub test 1a: Auto trigger initial OI scrub when server mounts ========================================================== 23:06:52 (1717556812) [ 228.517639] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556801/real 1717556801] req@ffff8800a6cfc700 x1800988768913344/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556817 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 228.517664] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556801/real 1717556801] req@ffff8800a6cff100 x1800988768913216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556817 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 228.533095] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556801/real 1717556801] req@ffff8800a6cfdc00 x1800988768913152/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556817 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 230.328813] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556801/real 1717556801] req@ffff8800a6cfd880 x1800988768913280/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556817 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 230.377467] Lustre: Failing over lustre-MDT0000 [ 230.420796] Lustre: server umount lustre-MDT0000 complete [ 230.420865] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 230.420936] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 230.429911] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 230.430029] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 230.437474] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 230.437651] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 230.437653] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 230.437749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 231.520077] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 231.520966] LustreError: 21976:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556820 with bad export cookie 11797782520123739914 [ 231.521101] Lustre: Failing over lustre-MDT0001 [ 231.521112] LustreError: 21978:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556820 with bad export cookie 11797782520123739914 [ 231.521137] LustreError: 21977:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556820 with bad export cookie 11797782520123739914 [ 231.521156] LustreError: 21978:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556820 with bad export cookie 11797782520123739914 [ 231.521201] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 231.659946] LustreError: 23131:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556820 with bad export cookie 11797782520123739914 [ 233.813218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 241.428825] Lustre: server umount lustre-MDT0001 complete [ 241.514783] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8800a6c49c00 x1800988768959232/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 242.319996] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 245.005615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 245.073805] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.073844] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 245.073895] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 246.176412] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 247.118308] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.121747] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 251.100737] Lustre: lustre-MDT0000: Denying connection for new client 27ff5fe0-059c-4f4a-8454-76134743968e (at 192.168.202.42@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 251.112598] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556824/real 1717556824] req@ffff8800a6c4a300 x1800988768958912/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556840 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 252.112613] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556824/real 1717556824] req@ffff8800a6c4b800 x1800988768958784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556840 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 252.120561] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556825/real 1717556825] req@ffff8800a6c48380 x1800988768958976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556841 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 253.236156] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556825/real 1717556825] req@ffff8800a6c4b480 x1800988768959040/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556841 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 253.300380] Lustre: Failing over lustre-MDT0000 [ 255.093143] Lustre: server umount lustre-MDT0000 complete [ 255.093156] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 255.093391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 255.096340] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 255.096414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 255.096582] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 255.096606] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 255.135346] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 255.838651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 255.865912] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 256.923640] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 256.929784] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556830/real 1717556830] req@ffff8800a6c4aa00 x1800988768959360/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556846 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 256.950092] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556830/real 1717556830] req@ffff8800a6c49500 x1800988768959296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556846 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 256.950165] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556829/real 1717556829] req@ffff8800a6c49880 x1800988768959168/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556845 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 257.026336] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556829/real 1717556829] req@ffff8800a6c48a80 x1800988768959104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556845 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 259.966657] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 260.949123] Lustre: DEBUG MARKER: == sanity-scrub test 1b: Trigger OI scrub when MDT mounts for OI files remove/recreate case ========================================================== 23:07:28 (1717556848) [ 263.438292] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 275.603676] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 290.423120] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 290.492386] Lustre: Failing over lustre-MDT0000 [ 290.997410] Lustre: server umount lustre-MDT0000 complete [ 290.997490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.997614] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.997795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.998164] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 291.035753] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 291.035875] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 291.046847] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 291.676291] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 291.677303] LustreError: 21977:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556881 with bad export cookie 11797782520123768782 [ 291.677392] Lustre: Failing over lustre-MDT0001 [ 291.677424] LustreError: 21978:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556881 with bad export cookie 11797782520123768782 [ 291.677446] LustreError: 23131:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556881 with bad export cookie 11797782520123768782 [ 291.677468] LustreError: 21978:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556881 with bad export cookie 11797782520123768782 [ 291.677519] LustreError: 24335:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556881 with bad export cookie 11797782520123768782 [ 291.799595] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 293.046955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 294.570443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 297.416505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 302.004907] Lustre: server umount lustre-MDT0001 complete [ 302.088502] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88011ea72300 x1800988769032128/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 303.119053] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 306.206110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 306.311813] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 306.312107] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.312701] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.366437] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 308.795665] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 308.801858] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 312.353691] Lustre: lustre-MDT0000: Denying connection for new client b44967d4-37f0-42aa-85cc-e5f25776f8aa (at 192.168.202.42@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 312.385697] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556885/real 1717556885] req@ffff88011ea71180 x1800988769031808/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556901 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 313.358752] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556885/real 1717556885] req@ffff88011ea72a00 x1800988769031680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556901 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 313.364491] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556886/real 1717556886] req@ffff88011ea73480 x1800988769031936/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556902 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 316.142062] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556886/real 1717556886] req@ffff88011ea71c00 x1800988769031872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556902 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 317.356625] Lustre: DEBUG MARKER: == sanity-scrub test 1c: Auto detect kinds of OI file(s) removed/recreated cases ========================================================== 23:08:24 (1717556904) [ 318.325661] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556890/real 1717556890] req@ffff88011ea72d80 x1800988769032000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556906 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 318.558659] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556891/real 1717556891] req@ffff88011ea70380 x1800988769032192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556907 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 318.568656] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556891/real 1717556891] req@ffff88011ea72680 x1800988769032256/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556907 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 323.474025] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556890/real 1717556890] req@ffff88011ea73800 x1800988769032064/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556906 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 323.531259] Lustre: Failing over lustre-MDT0000 [ 324.461125] Lustre: server umount lustre-MDT0000 complete [ 324.461799] LustreError: 21976:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556913 with bad export cookie 11797782520123795543 [ 324.461959] Lustre: Failing over lustre-MDT0001 [ 324.462003] LustreError: 25591:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556913 with bad export cookie 11797782520123795543 [ 324.462007] LustreError: 21978:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556913 with bad export cookie 11797782520123795543 [ 324.462009] LustreError: 25591:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556913 with bad export cookie 11797782520123795543 [ 324.462043] LustreError: 21977:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556913 with bad export cookie 11797782520123795543 [ 324.570149] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 325.923693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 327.433298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 330.186257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 330.205490] Lustre: server umount lustre-MDT0001 complete [ 334.462049] Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64 [ 334.616551] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88012e99f480 x1800988769084224/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 334.634813] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 334.634846] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 334.634859] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 334.634895] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 334.635019] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 334.635049] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 335.786564] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 338.956161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 338.979613] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 339.068968] Lustre: lustre-MDT0001: invalid oi count 63, remove them, then set it to 64 [ 339.106519] mount.lustre (11057) used greatest stack depth: 9920 bytes left [ 340.210079] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 342.076590] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 342.083889] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556915/real 1717556915] req@ffff88012e99ce00 x1800988769083456/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556931 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 342.096990] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556915/real 1717556915] req@ffff88012e99c700 x1800988769083648/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556931 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 342.108053] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556915/real 1717556915] req@ffff88012e99ea00 x1800988769083584/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556931 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 344.101017] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556915/real 1717556915] req@ffff88012e99d880 x1800988769083520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556931 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 345.102599] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 345.104892] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556918/real 1717556918] req@ffff88006fe4d180 x1800988769083904/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556934 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 345.112565] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556918/real 1717556918] req@ffff88006fe4f100 x1800988769083712/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556934 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 345.128617] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556918/real 1717556918] req@ffff88006fe4d500 x1800988769083840/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556934 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 346.342619] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556918/real 1717556918] req@ffff88006fe4f800 x1800988769083776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556934 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 350.124693] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 350.129851] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556923/real 1717556923] req@ffff88006fe4ca80 x1800988769084032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556939 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 350.140589] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556923/real 1717556923] req@ffff88006fe4d880 x1800988769084096/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556939 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 350.141602] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556923/real 1717556923] req@ffff88006fe4fb80 x1800988769083968/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556939 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 358.474344] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556923/real 1717556923] req@ffff88006fe4ed80 x1800988769084160/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556939 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 372.013950] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 372.091354] Lustre: Failing over lustre-MDT0000 [ 373.355186] Lustre: server umount lustre-MDT0000 complete [ 373.356603] LustreError: 23131:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556962 with bad export cookie 11797782520123823375 [ 373.356683] LustreError: 25591:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556962 with bad export cookie 11797782520123823375 [ 373.356697] LustreError: 21978:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556962 with bad export cookie 11797782520123823375 [ 373.356700] LustreError: 25591:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556962 with bad export cookie 11797782520123823375 [ 373.356767] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 373.356864] Lustre: Failing over lustre-MDT0001 [ 373.480975] LustreError: 21977:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717556962 with bad export cookie 11797782520123823375 [ 375.086655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 377.641009] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 381.407633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 381.432372] Lustre: server umount lustre-MDT0001 complete [ 383.356843] Lustre: lustre-MDT0000: invalid oi count 58, remove them, then set it to 64 [ 383.478911] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88012e512a00 x1800988769148352/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 384.201851] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 386.361101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 386.373309] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 386.438031] Lustre: lustre-MDT0001: invalid oi count 58, remove them, then set it to 64 [ 386.438090] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.438091] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.438624] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.463107] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 387.283663] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 390.462611] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 390.468554] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556963/real 1717556963] req@ffff88009fee8a80 x1800988769147776/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556979 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 390.480234] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556963/real 1717556963] req@ffff88009feea680 x1800988769147584/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556979 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 390.489194] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556963/real 1717556963] req@ffff88009fee8000 x1800988769147648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556979 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 391.476977] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556963/real 1717556963] req@ffff88009fee8e00 x1800988769147712/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556979 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 392.480186] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 394.450618] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 394.485655] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556967/real 1717556967] req@ffff88009fee9500 x1800988769147904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556983 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 394.502638] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556967/real 1717556967] req@ffff88009feead80 x1800988769148032/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556983 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 394.516291] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556967/real 1717556967] req@ffff88009feeb100 x1800988769147968/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556983 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 398.620582] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556967/real 1717556967] req@ffff88009fee9180 x1800988769147840/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556983 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 398.620587] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556972/real 1717556972] req@ffff880092a61180 x1800988769148160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556988 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 398.620602] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556972/real 1717556972] req@ffff880092a63100 x1800988769148224/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556988 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 399.516589] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556972/real 1717556972] req@ffff8800842e5f80 x1800988769148096/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556988 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 403.830232] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717556972/real 1717556972] req@ffff880092a60000 x1800988769148288/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717556988 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 416.852802] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 416.907667] Lustre: Failing over lustre-MDT0000 [ 417.852863] Lustre: server umount lustre-MDT0000 complete [ 417.854215] LustreError: 21976:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557007 with bad export cookie 11797782520123850857 [ 417.861081] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 417.965528] Lustre: Failing over lustre-MDT0001 [ 419.149476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 420.878377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 423.760698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 423.772009] Lustre: server umount lustre-MDT0001 complete [ 427.861290] Lustre: lustre-MDT0000: invalid oi count 56, remove them, then set it to 64 [ 427.982374] Lustre: Evicted from MGS (at 192.168.202.142@tcp) after server handle changed from 0x0 to 0xa3ba24b1d94bebc3 [ 428.790508] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 430.952306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 430.963748] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 431.028784] Lustre: lustre-MDT0001: invalid oi count 56, remove them, then set it to 64 [ 431.028859] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 431.028882] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 431.028902] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 431.028929] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 431.879390] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 436.053073] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 436.682671] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 437.052578] Lustre: DEBUG MARKER: == sanity-scrub test 2: Trigger OI scrub when MDT mounts for backup/restore case ========================================================== 23:10:25 (1717557025) [ 437.052586] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557010/real 1717557010] req@ffff880092a0f100 x1800988769211200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557026 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 437.072708] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557010/real 1717557010] req@ffff880092a0f800 x1800988769211328/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557026 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 437.074604] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557010/real 1717557010] req@ffff880092a0c700 x1800988769211136/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557026 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 439.052663] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557010/real 1717557010] req@ffff880092a0f480 x1800988769211264/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557026 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 439.068590] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557012/real 1717557012] req@ffff880092a0df80 x1800988769211584/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557028 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 439.074634] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557012/real 1717557012] req@ffff880092a0ca80 x1800988769211456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557028 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 439.093635] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557012/real 1717557012] req@ffff880092a0d180 x1800988769211520/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557028 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 439.584722] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557012/real 1717557012] req@ffff880092a0fb80 x1800988769211392/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557028 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 444.060572] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 444.076582] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557017/real 1717557017] req@ffff88009bc16300 x1800988769211776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557033 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 444.077559] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557017/real 1717557017] req@ffff88009bc15f80 x1800988769211712/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557033 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 444.077575] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557017/real 1717557017] req@ffff88009bc15c00 x1800988769211904/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557033 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 451.441611] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557017/real 1717557017] req@ffff88009bc17b80 x1800988769211840/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557033 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 464.567782] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 464.617452] Lustre: Failing over lustre-MDT0000 [ 465.702132] Lustre: server umount lustre-MDT0000 complete [ 465.703815] LustreError: 23131:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557055 with bad export cookie 11797782520123878339 [ 465.703828] LustreError: 21978:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557055 with bad export cookie 11797782520123878339 [ 465.703926] LustreError: 24335:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557055 with bad export cookie 11797782520123878339 [ 465.703996] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 465.853726] Lustre: Failing over lustre-MDT0001 [ 467.398111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 470.069586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 470.352729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 472.700950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 475.146096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 475.427049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 478.600328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 478.609614] Lustre: server umount lustre-MDT0001 complete [ 481.708659] Lustre: lustre-MDT0000: reset Object Index mappings [ 481.708667] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557055/real 1717557055] req@ffff88012e510e00 x1800988769265216/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557071 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 481.708673] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557055/real 1717557055] req@ffff88012e511180 x1800988769265280/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557071 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 481.708708] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557055/real 1717557055] req@ffff88012e511c00 x1800988769265152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557071 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 481.708712] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 481.708737] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 481.708756] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557055/real 1717557055] req@ffff88012e510000 x1800988769265088/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557071 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 481.781386] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 486.708735] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 486.709647] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557060/real 1717557060] req@ffff88012e510380 x1800988769265344/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557076 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 486.709684] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557060/real 1717557060] req@ffff88012e512a00 x1800988769265472/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557076 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 486.709705] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557060/real 1717557060] req@ffff88012e511f80 x1800988769265536/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557076 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 490.785366] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557060/real 1717557060] req@ffff88012e511880 x1800988769265408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557076 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 490.949828] Lustre: Evicted from MGS (at 192.168.202.142@tcp) after server handle changed from 0x0 to 0xa3ba24b1d94c571d [ 491.751658] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 491.869218] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557065/real 1717557065] req@ffff88012e513100 x1800988769265664/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557081 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 491.952603] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 491.958780] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557065/real 1717557065] req@ffff88012e512680 x1800988769265792/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557081 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 491.959429] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557065/real 1717557065] req@ffff88012e511500 x1800988769265728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557081 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 491.959613] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 491.959689] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 491.964386] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.727488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 494.731172] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557065/real 1717557065] req@ffff88009101bb80 x1800988769265856/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557081 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 494.796635] Lustre: lustre-MDT0001: reset Object Index mappings [ 495.800123] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 496.796578] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 496.801546] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557070/real 1717557070] req@ffff880091018700 x1800988769265920/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557086 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 496.810573] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557070/real 1717557070] req@ffff88009101b100 x1800988769265984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557086 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 496.810582] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557070/real 1717557070] req@ffff88009101b800 x1800988769266112/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557086 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 497.113970] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557070/real 1717557070] req@ffff880091019f80 x1800988769266048/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557086 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 497.123809] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 497.800651] Lustre: lustre-MDT0000: Denying connection for new client 06421c9e-3532-428f-9252-6f3f369f762d (at 192.168.202.42@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 503.890669] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557071/real 1717557071] req@ffff880091019500 x1800988769266304/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557087 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 510.525490] Lustre: DEBUG MARKER: == sanity-scrub test 4a: Auto trigger OI scrub if bad OI mapping was found (1) ========================================================== 23:11:32 (1717557092) [ 510.589157] Lustre: Failing over lustre-MDT0000 [ 511.595972] Lustre: server umount lustre-MDT0000 complete [ 511.597011] LustreError: 21977:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557101 with bad export cookie 11797782520123905821 [ 511.597122] Lustre: Failing over lustre-MDT0001 [ 511.597130] LustreError: 21976:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557101 with bad export cookie 11797782520123905821 [ 511.597144] LustreError: 23131:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557101 with bad export cookie 11797782520123905821 [ 511.597160] LustreError: 21976:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557101 with bad export cookie 11797782520123905821 [ 511.597179] LustreError: 21976:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557101 with bad export cookie 11797782520123905821 [ 511.704335] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 513.427504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 516.603271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 516.876937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 519.419162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 522.481022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 522.802156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 526.554152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 526.563028] Lustre: server umount lustre-MDT0001 complete [ 530.612677] Lustre: lustre-MDT0000: reset Object Index mappings [ 530.612682] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557104/real 1717557104] req@ffff88009c06c380 x1800988769312832/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 530.612716] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557104/real 1717557104] req@ffff88009c06d180 x1800988769312704/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 530.612772] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.612800] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557104/real 1717557104] req@ffff88009c06c000 x1800988769312768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 530.612810] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557104/real 1717557104] req@ffff88009c06fb80 x1800988769312896/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 530.612815] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.671764] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.612793] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.612807] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557105/real 1717557105] req@ffff880129dbad80 x1800988769313152/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557121 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 531.612837] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557105/real 1717557105] req@ffff880129dbaa00 x1800988769313088/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557121 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 531.612869] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557105/real 1717557105] req@ffff880129db8e00 x1800988769312960/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557121 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 536.612704] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557105/real 1717557105] req@ffff880129dbb480 x1800988769313024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557121 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 536.612709] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557110/real 1717557110] req@ffff880129dbbb80 x1800988769313216/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557126 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 536.652719] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557110/real 1717557110] req@ffff880129dbb800 x1800988769313280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557126 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 536.674668] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557110/real 1717557110] req@ffff880129db8700 x1800988769313408/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557126 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 536.675960] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557110/real 1717557110] req@ffff880129db9c00 x1800988769313344/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557126 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 536.676020] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88009bc24380 x1800988769314112/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 536.676078] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88009bc25500 x1800988769314176/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 536.801769] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88009bc25880 x1800988769314240/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 537.815159] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.812351] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 540.812362] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.812459] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.839627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 540.848427] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.943305] Lustre: lustre-MDT0001: reset Object Index mappings [ 542.072299] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 542.667601] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 542.952652] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557116/real 1717557116] req@ffff8800a6ef5880 x1800988769313728/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557132 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 542.964630] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557116/real 1717557116] req@ffff8800a6ef6680 x1800988769313536/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557132 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 542.969395] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557116/real 1717557116] req@ffff8800a6ef7100 x1800988769313600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557132 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 543.515419] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557116/real 1717557116] req@ffff8800a6ef6a00 x1800988769313664/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557132 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 543.530288] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 547.008683] Lustre: lustre-MDT0000: Denying connection for new client 1a3a0a8a-1edb-4d3b-8fb6-b56bfaf895e5 (at 192.168.202.42@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 549.521203] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557120/real 1717557120] req@ffff880129db8380 x1800988769313920/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557136 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 550.569073] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200004282:0x1:0x0]/32043: rc = 0 [ 561.707074] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240003ab1:0x1:0x0]/32004 with flags 0x52: rc = 0 [ 568.461036] Lustre: DEBUG MARKER: == sanity-scrub test 4b: Auto trigger OI scrub if bad OI mapping was found (2) ========================================================== 23:12:30 (1717557150) [ 568.519714] Lustre: Failing over lustre-MDT0000 [ 569.495011] Lustre: server umount lustre-MDT0000 complete [ 569.495893] LustreError: 21976:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557158 with bad export cookie 11797782520123933219 [ 569.495924] LustreError: 21977:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557158 with bad export cookie 11797782520123933219 [ 569.495949] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 569.495992] LustreError: 21977:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557158 with bad export cookie 11797782520123933219 [ 569.604492] Lustre: Failing over lustre-MDT0001 [ 570.981373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 573.498357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 573.750662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 575.899749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 578.467466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 578.849018] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 582.259822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 582.266839] Lustre: server umount lustre-MDT0001 complete [ 587.500593] Lustre: lustre-MDT0000: reset Object Index mappings [ 587.500630] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557160/real 1717557160] req@ffff88012c418e00 x1800988769362240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557176 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 587.500643] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557160/real 1717557160] req@ffff88012c41b800 x1800988769362176/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557176 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 587.500657] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557160/real 1717557160] req@ffff88012c41a300 x1800988769362368/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557176 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 587.500666] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 587.500681] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 587.500688] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557160/real 1717557160] req@ffff88012c419f80 x1800988769362304/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557176 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 587.541412] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.544708] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 590.500722] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557163/real 1717557163] req@ffff880129db9f80 x1800988769362496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557179 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 590.500730] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557163/real 1717557163] req@ffff880129db8a80 x1800988769362560/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557179 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 590.500734] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557163/real 1717557163] req@ffff880129dbad80 x1800988769362624/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557179 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 593.544850] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557163/real 1717557163] req@ffff880129dba300 x1800988769362432/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557179 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 593.702981] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88012c418e00 x1800988769363584/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 594.543580] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.618465] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557168/real 1717557168] req@ffff880129dbaa00 x1800988769362816/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557184 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 594.709599] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 594.722611] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557168/real 1717557168] req@ffff880129db9180 x1800988769362880/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557184 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 594.732143] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557168/real 1717557168] req@ffff880129db8700 x1800988769362688/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557184 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 596.719995] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557168/real 1717557168] req@ffff880129dbbb80 x1800988769362752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557184 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 596.720200] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 597.321875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 597.329204] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 597.423232] Lustre: lustre-MDT0001: reset Object Index mappings [ 598.284547] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 598.440207] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 599.708590] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 599.724651] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557173/real 1717557173] req@ffff880129dbb480 x1800988769363136/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557189 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 600.441660] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557173/real 1717557173] req@ffff880129dbb100 x1800988769363008/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557189 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 600.448688] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557173/real 1717557173] req@ffff880129db9880 x1800988769363200/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557189 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 600.643466] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557173/real 1717557173] req@ffff880129dba680 x1800988769363072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557189 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 600.643467] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200004a52:0x1:0x0]/64001: rc = 0 [ 603.436690] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557176/real 1717557176] req@ffff880092a63800 x1800988769363328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557192 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworLustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240004281:0x1:0x0]/96020 with flags 0x52: rc = 0 [ 696.144739] Lustre: DEBUG MARKER: == sanity-scrub test 4c: Auto trigger OI scrub if bad OI mapping was found (3) ========================================================== 23:14:44 (1717557284)LustreError: 25013:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1281:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88009cafd880 x1800988769427648/t0(0) o104->lustre-OST0001@192.168.202.42@tcp:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 [ 701.371429] LustreError: 22679:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1281:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88012e511c00 x1800988769428608/t0(0) o104->lustre-OST0000@192.168.202.42@tcp:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295Lustre: Failing over lustre-MDT0000 [ 705.835727] Lustre: server umount lustre-MDT0000 completeLustreError: 23131:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557296 with bad export cookie 11797782520123960722[ 707.045848] Lustre: Failing over lustre-MDT0001LustreError: 24335:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557296 with bad export cookie 11797782520123960722LustreError: 25591:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557296 with bad export cookie 11797782520123960722LustreError: 25591:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557296 with bad export cookie 11797782520123960722LustreError: 21977:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557296 with bad export cookie 11797782520123960722LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: server umount lustre-MDT0001 complete[ 709.013021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 712.156343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 712.455932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 715.293668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 717.821142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 718.073190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 721.898314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 721.910025] Lustre: lustre-MDT0000: reset Object Index mappings [ 725.052648] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557298/real 1717557298] req@ffff8800935faa00 x1800988769437184/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557314 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworLustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557298/real 1717557298] req@ffff8800935f9880 x1800988769437120/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557314 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworLustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557298/real 1717557298] req@ffff8800935f8700 x1800988769436992/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557314 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworLustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 725.052735] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 725.052746] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557298/real 1717557298] req@ffff8800935f9500 x1800988769437056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557314 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 725.100573] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 729.052637] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 729.052717] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557302/real 1717557302] req@ffff8800935f9c00 x1800988769437440/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557318 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 729.052726] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557302/real 1717557302] req@ffff8800935f8380 x1800988769437376/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557318 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 729.105644] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557302/real 1717557302] req@ffff8800935fbb80 x1800988769437312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557318 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 732.106040] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557302/real 1717557302] req@ffff8800935fa680 x1800988769437248/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557318 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 732.252862] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8800a6c7b100 x1800988769438400/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 733.052698] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 733.059994] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557306/real 1717557306] req@ffff8800935f9180 x1800988769437632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557322 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 733.091705] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557306/real 1717557306] req@ffff8800a6c79f80 x1800988769437696/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557322 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 733.285650] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557306/real 1717557306] req@ffff8800935fb480 x1800988769437568/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557322 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 733.433636] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557306/real 1717557306] req@ffff8800a6c7b800 x1800988769437760/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557322 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 735.282161] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 735.282271] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 735.282458] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 736.492973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 736.502575] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 736.599957] Lustre: lustre-MDT0001: reset Object Index mappings [ 737.607656] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 737.621596] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557311/real 1717557311] req@ffff8800a6c7ad80 x1800988769437888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557327 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 737.621603] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557311/real 1717557311] req@ffff8800a6c7a300 x1800988769438016/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557327 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 737.666281] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557311/real 1717557311] req@ffff8800a6c7bb80 x1800988769437952/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557327 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 738.281596] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 739.083529] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557311/real 1717557311] req@ffff8800a6c79880 x1800988769437824/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557327 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 739.091635] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 741.284658] Lustre: lustre-MDT0000: Denying connection for new client c25b67e8-df6a-484e-866a-f38cf379385b (at 192.168.202.42@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 745.565169] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557314/real 1717557314] req@ffff88012b8d9f80 x1800988769438272/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557330 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 748.616283] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200005222:0x1:0x0]/32035: rc = 0 [ 809.101669] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240004a51:0x1:0x0]/32001 with flags 0x52: rc = 0 [ 816.819215] Lustre: DEBUG MARKER: == sanity-scrub test 4d: FID in LMA mismatch with object FID won't block create ========================================================== 23:16:37 (1717557397) [ 821.697519] Lustre: DEBUG MARKER: == sanity-scrub test 4e: FID reuse can be fixed ========== 23:16:45 (1717557405) [ 836.772864] Lustre: DEBUG MARKER: == sanity-scrub test 5: OI scrub state machine =========== 23:16:50 (1717557410) [ 836.772951] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 836.772972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 836.773019] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.698196] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 839.656959] Lustre: server umount lustre-MDT0000 complete [ 839.657642] LustreError: 24335:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557429 with bad export cookie 11797782520124004941 [ 839.657662] LustreError: 21976:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557429 with bad export cookie 11797782520124004941 [ 839.657686] LustreError: 21977:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557429 with bad export cookie 11797782520124004941 [ 839.657689] LustreError: 21977:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557429 with bad export cookie 11797782520124004941 [ 839.657710] LustreError: 21976:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557429 with bad export cookie 11797782520124004941 [ 841.780911] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 841.780913] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 841.780938] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 841.780972] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 844.788870] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 844.789018] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 845.747396] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 852.689420] Lustre: server umount lustre-MDT0001 complete [ 859.653579] Lustre: server umount lustre-OST0000 complete [ 860.852261] Lustre: server umount lustre-OST0001 complete [ 861.954086] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 864.333417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 866.309837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 867.969794] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 867.971963] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 867.975150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 870.224794] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 870.229509] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 874.295035] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local[ 877.335911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 877.358617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.511988] Lustre: lustre-MDT0000: new disk, initializingLustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all[ 882.890138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 882.916078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 883.004340] Lustre: lustre-MDT0001: new disk, initializingLustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 888.635789] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 888.641875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 888.669717] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 888.675882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 888.754228] Lustre: lustre-OST0000: new disk, initializing [ 890.319391] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck allLDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 894.703744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 894.726204] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 894.730553] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 894.759826] Lustre: lustre-OST0001: new disk, initializing [ 896.530623] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck allLustre: DEBUG MARKER: Using TIMEOUT=20 [ 913.932422] Lustre: Failing over lustre-MDT0000Lustre: server umount lustre-MDT0000 complete [ 915.230282] LustreError: 29619:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557504 with bad export cookie 11797782520124098965Lustre: Failing over lustre-MDT0001 [ 915.231753] LustreError: 28141:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557504 with bad export cookie 11797782520124098965 [ 915.231794] LustreError: 28143:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557504 with bad export cookie 11797782520124098965 [ 915.231807] LustreError: 28141:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557504 with bad export cookie 11797782520124098965 [ 915.231846] LustreError: 28143:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557504 with bad export cookie 11797782520124098965 [ 915.390316] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 917.440455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 920.554641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 920.846573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 923.705351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 926.589145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 926.857350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 929.589727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 929.594528] Lustre: server umount lustre-MDT0001 complete [ 933.231639] Lustre: lustre-MDT0000: reset Object Index mappings [ 933.234345] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557506/real 1717557506] req@ffff88009f387100 x1800988769570368/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557522 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 933.244610] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 933.244620] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557506/real 1717557506] req@ffff88009f384380 x1800988769570240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557522 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 933.244624] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557506/real 1717557506] req@ffff88009f384e00 x1800988769570176/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557522 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 933.244634] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 933.270030] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 933.273549] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557506/real 1717557506] req@ffff88009f386680 x1800988769570304/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557522 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 937.244641] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 937.244648] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557510/real 1717557510] req@ffff8801317cf800 x1800988769570624/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557526 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 937.244653] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557510/real 1717557510] req@ffff8801317cd500 x1800988769570496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557526 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 937.280586] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557510/real 1717557510] req@ffff8801317cc380 x1800988769570432/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557526 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 940.280723] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557510/real 1717557510] req@ffff8801317cc700 x1800988769570560/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557526 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 940.373226] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8801317cf800 x1800988769571584/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 941.099639] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 941.244567] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 941.248493] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557514/real 1717557514] req@ffff88009f387b80 x1800988769570816/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557530 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 941.382609] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557514/real 1717557514] req@ffff88009f387800 x1800988769570752/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557530 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 941.388574] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557514/real 1717557514] req@ffff88009f385c00 x1800988769570880/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557530 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 943.208739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 943.214090] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557514/real 1717557514] req@ffff88009f385880 x1800988769570944/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557530 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 943.283717] Lustre: lustre-MDT0001: reset Object Index mappings [ 943.973410] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 944.294944] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 946.289605] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 946.292315] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557519/real 1717557519] req@ffff8801317ce680 x1800988769571136/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557535 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 946.297638] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557519/real 1717557519] req@ffff8801317cd880 x1800988769571200/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557535 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 946.396586] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557519/real 1717557519] req@ffff8801317cce00 x1800988769571008/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557535 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 946.436083] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557519/real 1717557519] req@ffff8801317cdc00 x1800988769571072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557535 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 949.297603] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200000402:0x1:0x0]/32009: rc = 0 [ 949.478426] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557522/real 1717557522] req@ffff8801317cf480 x1800988769571392/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557538 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 956.095023] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240000402:0x1:0x0]/64025 with flags 0x52: rc = 0 [ 956.142230] Lustre: Failing over lustre-MDT0000 [ 957.090954] Lustre: server umount lustre-MDT0000 complete [ 957.091728] LustreError: 28142:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557546 with bad export cookie 11797782520124117886 [ 957.091770] Lustre: Failing over lustre-MDT0001 [ 957.091778] LustreError: 29095:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557546 with bad export cookie 11797782520124117886 [ 957.091814] LustreError: 28141:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557546 with bad export cookie 11797782520124117886 [ 957.183882] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 960.347678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 967.092067] Lustre: server umount lustre-MDT0001 complete [ 967.216184] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8800a72b7b80 x1800988769584768/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 968.209816] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 970.967666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 971.032402] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 971.032411] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 971.032438] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 971.032565] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 971.049233] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 971.778923] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 973.222142] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 973.226763] Lustre: Failing over lustre-MDT0000 [ 973.228745] LustreError: 6089:0:(/home/green/git/lustre-release/lustre/osp/osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 973.235533] LustreError: 6089:0:(/home/green/git/lustre-release/lustre/lod/lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 973.240805] LustreError: 6089:0:(/home/green/git/lustre-release/lustre/lod/lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 973.240869] Lustre: 6090:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 973.254665] LustreError: 7572:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2927:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 973.259718] Lustre: 6090:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 973.271323] Lustre: 6090:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 973.276502] LustreError: 6090:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1281:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88012c41b100 x1800988769592384/t0(0) o1000->lustre-MDT0001-osp-MDT0000@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'tgt_recover_0.0' uid:0 gid:0 [ 973.284136] LustreError: 6090:0:(/home/green/git/lustre-release/lustre/osp/osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 973.300963] LustreError: 6090:0:(/home/green/git/lustre-release/lustre/lod/lod_dev.c:365:lod_sub_recreate_llog()) lustre-MDT0000-mdtlov: can't access update_log: rc = -5 [ 973.356723] Lustre: 6090:0:(/home/green/git/lustre-release/lustre/mdt/mdt_handler.c:7962:mdt_postrecov()) lustre-MDT0000: auto trigger paused LFSCK failed: rc = -6 [ 974.385018] Lustre: server umount lustre-MDT0000 complete [ 974.386082] LustreError: 29095:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557563 with bad export cookie 11797782520124119664 [ 974.386092] LustreError: 28143:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557563 with bad export cookie 11797782520124119664 [ 974.386126] LustreError: 29619:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557563 with bad export cookie 11797782520124119664 [ 974.386171] LustreError: 28143:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557563 with bad export cookie 11797782520124119664 [ 974.386254] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 974.386267] Lustre: Failing over lustre-MDT0001 [ 974.390126] LustreError: 28143:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557563 with bad export cookie 11797782520124119664 [ 974.390154] LustreError: 6942:0:(/home/green/git/lustre-release/lustre/osp/osp_object.c:637:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000009:0x0:0x0]: rc = -5 [ 974.390160] LustreError: 6942:0:(/home/green/git/lustre-release/lustre/lod/lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't get id from catalogs: rc = -5 [ 974.509522] LustreError: 6942:0:(/home/green/git/lustre-release/lustre/lod/lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0000-osp-MDT0001: get update log duration 3, retries 0, failed: rc = -5 [ 975.385609] Lustre: server umount lustre-MDT0001 complete [ 975.385613] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557548/real 1717557548] req@ffff88012e56b480 x1800988769584256/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557564 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 975.385681] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557548/real 1717557548] req@ffff88012e569180 x1800988769584320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557564 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 975.385698] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557548/real 1717557548] req@ffff88012e568700 x1800988769584448/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557564 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 977.389330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 978.068707] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557548/real 1717557548] req@ffff88012e56a300 x1800988769584384/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557564 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 978.084774] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557551/real 1717557551] req@ffff88012e56bb80 x1800988769584512/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557567 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 978.385725] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557551/real 1717557551] req@ffff88012e568000 x1800988769584576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557567 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 978.405724] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557551/real 1717557551] req@ffff88012e569c00 x1800988769584704/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557567 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 983.076619] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557551/real 1717557551] req@ffff88012e569f80 x1800988769584640/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557567 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 983.080624] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557556/real 1717557556] req@ffff8800a72b4a80 x1800988769584896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557572 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 983.421705] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557556/real 1717557556] req@ffff8800a72b4700 x1800988769584832/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557572 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 983.436135] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557556/real 1717557556] req@ffff8800a72b5180 x1800988769585024/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557572 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 984.068819] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557556/real 1717557556] req@ffff8800a72b7100 x1800988769584960/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557572 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 984.180325] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88012c41b100 x1800988769593536/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 985.036401] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 986.191164] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 986.191173] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 986.191215] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 986.191277] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 986.191375] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 987.458753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 987.541705] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 988.624393] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 991.540763] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 991.560602] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557565/real 1717557565] req@ffff8800a6c28e00 x1800988769593024/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557581 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 992.565671] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557565/real 1717557565] req@ffff8800a6c29f80 x1800988769593152/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557581 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 994.458912] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 994.461386] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200000402:0x45:0x0]/220: rc = 0 [ 994.569615] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240000402:0x1:0x0]/64025 with flags 0x52: rc = 0 [ 994.584666] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557568/real 1717557568] req@ffff8800a6c2b480 x1800988769593280/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557584 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 996.590572] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557568/real 1717557568] req@ffff8800a6c29880 x1800988769593344/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557584 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 996.615640] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557570/real 1717557570] req@ffff8800a6c2b100 x1800988769593408/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557586 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 997.956670] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557570/real 1717557570] req@ffff8800a6c2ad80 x1800988769593472/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557586 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 999.945559] Lustre: DEBUG MARKER: == sanity-scrub test 6: OI scrub resumes from last checkpoint ========================================================== 23:19:46 (1717557586) [ 999.949326] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557573/real 1717557573] req@ffff88012c41b480 x1800988769593600/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557589 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1005.314297] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557573/real 1717557573] req@ffff88012c419c00 x1800988769593664/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557589 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1005.405542] Lustre: Failing over lustre-MDT0000 [ 1006.663183] Lustre: server umount lustre-MDT0000 complete [ 1006.664400] LustreError: 29619:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557596 with bad export cookie 11797782520124120812 [ 1006.664618] Lustre: Failing over lustre-MDT0001 [ 1006.664621] LustreError: 28142:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557596 with bad export cookie 11797782520124120812 [ 1006.664711] LustreError: 28141:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557596 with bad export cookie 11797782520124120812 [ 1006.664714] LustreError: 28142:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557596 with bad export cookie 11797782520124120812 [ 1006.664763] LustreError: 29095:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557596 with bad export cookie 11797782520124120812 [ 1006.815884] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1008.647092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1011.249335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1011.510020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1013.898499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1017.341612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1017.704841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1021.855765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1021.867496] Lustre: server umount lustre-MDT0001 complete [ 1023.668637] Lustre: lustre-MDT0000: reset Object Index mappings [ 1023.668663] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557597/real 1717557597] req@ffff8800a72b4000 x1800988769639552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557613 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1023.668697] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557597/real 1717557597] req@ffff8800a72b4e00 x1800988769639488/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557613 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1023.668700] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557597/real 1717557597] req@ffff8800a72b4700 x1800988769639616/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557613 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1023.668733] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.668761] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.668781] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557597/real 1717557597] req@ffff8800a72b6680 x1800988769639680/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557613 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1023.742743] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1028.668734] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1028.668758] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557602/real 1717557602] req@ffff8800a72b7100 x1800988769639872/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1028.668835] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557602/real 1717557602] req@ffff8800a72b7800 x1800988769639744/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1028.746790] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557602/real 1717557602] req@ffff8800a72b6300 x1800988769639936/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1031.747171] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557602/real 1717557602] req@ffff8800a72b6d80 x1800988769639808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1031.913497] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff880073ad8e00 x1800988769640896/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1032.668630] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1032.668639] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557606/real 1717557606] req@ffff8800a72b7b80 x1800988769640064/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1032.874022] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557606/real 1717557606] req@ffff8800a72b5c00 x1800988769640256/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1032.916585] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1032.926604] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557606/real 1717557606] req@ffff8800a72b6a00 x1800988769640128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1033.924148] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557606/real 1717557606] req@ffff8800a72b5180 x1800988769640192/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1033.924238] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1033.924345] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1035.758832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1035.765546] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1035.842105] Lustre: lustre-MDT0001: reset Object Index mappings [ 1036.782968] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 1037.841614] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1037.847684] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557611/real 1717557611] req@ffff88009f387800 x1800988769640448/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1037.857605] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557611/real 1717557611] req@ffff8800a72b4a80 x1800988769640320/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1037.857685] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557611/real 1717557611] req@ffff8800a72b7480 x1800988769640384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1038.039303] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557611/real 1717557611] req@ffff88009f384000 x1800988769640512/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1038.050856] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1039.847681] Lustre: lustre-MDT0000: Denying connection for new client f7df05b1-c21c-4469-80de-0d487cb2112d (at 192.168.202.42@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 1044.222626] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557613/real 1717557613] req@ffff88009f384380 x1800988769640640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557629 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1047.260843] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200001b72:0x1:0x0]/32033: rc = 0 [ 1056.212611] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240001b71:0x1:0x0]/32004 with flags 0x52: rc = 0 [ 1057.231292] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200001b72:0x45:0x0]/170: rc = 0 [ 1064.809164] Lustre: lustre-MDT0001: trigger partial OI scrub for RPC inconsistency, checking FID [0x240001b71:0x44:0x0]/224: rc = 0 [ 1064.873214] Lustre: Failing over lustre-MDT0000 [ 1065.908969] Lustre: server umount lustre-MDT0000 complete [ 1065.909006] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1065.909015] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1065.909039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1065.909169] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1065.909219] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1065.909237] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1065.909327] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1065.960718] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1066.210045] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1066.210686] LustreError: 29095:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557655 with bad export cookie 11797782520124162238 [ 1066.210915] Lustre: Failing over lustre-MDT0001 [ 1066.210924] LustreError: 28142:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557655 with bad export cookie 11797782520124162238 [ 1066.210966] LustreError: 28143:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557655 with bad export cookie 11797782520124162238 [ 1066.210971] LustreError: 28143:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557655 with bad export cookie 11797782520124162238 [ 1066.210974] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1066.333448] LustreError: 28142:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557655 with bad export cookie 11797782520124162238 [ 1069.901364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1075.917059] Lustre: server umount lustre-MDT0001 complete [ 1076.039742] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88012df68000 x1800988769657344/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1076.877091] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.735160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1079.829735] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1079.829796] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 1079.829875] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1080.712985] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1084.853307] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1086.862640] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1086.862687] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557660/real 1717557660] req@ffff88012e568700 x1800988769657152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557676 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1089.151600] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557660/real 1717557660] req@ffff88012e56aa00 x1800988769657280/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557676 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1091.868697] Lustre: DEBUG MARKER: == sanity-scrub test 7: System is available during OI scrub scanning ========================================================== 23:21:17 (1717557677) [ 1091.868715] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557665/real 1717557665] req@ffff88012df69880 x1800988769657408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557681 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1092.496497] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557665/real 1717557665] req@ffff88012df69f80 x1800988769657472/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557681 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1106.995219] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 1125.931773] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1126.007767] Lustre: Failing over lustre-MDT0000 [ 1127.244955] Lustre: server umount lustre-MDT0000 complete [ 1127.245946] LustreError: 28141:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557716 with bad export cookie 11797782520124164128 [ 1127.245988] Lustre: Failing over lustre-MDT0001 [ 1127.246006] LustreError: 28143:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557716 with bad export cookie 11797782520124164128 [ 1127.246036] LustreError: 28142:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557716 with bad export cookie 11797782520124164128 [ 1127.246068] LustreError: 28143:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557716 with bad export cookie 11797782520124164128 [ 1127.246093] LustreError: 28143:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557716 with bad export cookie 11797782520124164128 [ 1127.384373] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1129.427137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1132.926055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1133.289997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1136.202384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1139.588338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1139.900388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1144.081580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1144.092368] Lustre: server umount lustre-MDT0001 complete [ 1147.260662] Lustre: lustre-MDT0000: reset Object Index mappings [ 1147.260679] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557720/real 1717557720] req@ffff880135507b80 x1800988769717760/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557736 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1147.260686] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557720/real 1717557720] req@ffff880135505880 x1800988769717888/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557736 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1147.260733] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557720/real 1717557720] req@ffff880135507800 x1800988769717824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557736 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1147.260743] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1147.260773] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1147.260803] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557720/real 1717557720] req@ffff880135505c00 x1800988769717952/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557736 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1147.331745] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1148.260714] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1148.260778] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557721/real 1717557721] req@ffff880135504a80 x1800988769718208/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557737 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1148.291084] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557721/real 1717557721] req@ffff880135504380 x1800988769718080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557737 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1148.334655] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557721/real 1717557721] req@ffff880135504700 x1800988769718144/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557737 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1152.350115] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557721/real 1717557721] req@ffff880135504000 x1800988769718016/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557737 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1152.439596] Lustre: Evicted from MGS (at 192.168.202.142@tcp) after server handle changed from 0x0 to 0xa3ba24b1d9511668 [ 1153.310566] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1153.349579] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557726/real 1717557726] req@ffff880135505500 x1800988769718400/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557742 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1153.359256] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557726/real 1717557726] req@ffff880135504e00 x1800988769718272/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557742 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1153.441684] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1153.449575] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557726/real 1717557726] req@ffff880135505180 x1800988769718336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557742 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1156.215993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1156.226901] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557726/real 1717557726] req@ffff880135507100 x1800988769718464/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557742 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1156.318459] Lustre: lustre-MDT0001: reset Object Index mappings [ 1157.332767] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 1157.377587] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1158.332719] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1158.347589] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557731/real 1717557731] req@ffff88012e56b100 x1800988769718656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557747 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1158.460632] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557731/real 1717557731] req@ffff88012e56a680 x1800988769718720/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557747 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1158.460724] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557731/real 1717557731] req@ffff88012e569180 x1800988769718784/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557747 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1159.592332] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557731/real 1717557731] req@ffff88012e568e00 x1800988769718592/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557747 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1160.632171] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200002b12:0x1:0x0]/32010: rc = 0 [ 1163.356799] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240002b11:0x1:0x0]/32029 with flags 0x52: rc = 0 [ 1170.594137] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557736/real 1717557736] req@ffff880135507800 x1800988769718848/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1181.583680] Lustre: DEBUG MARKER: == sanity-scrub test 8: Control OI scrub manually ======== 23:22:38 (1717557758) [ 1181.635654] Lustre: Failing over lustre-MDT0000 [ 1182.373099] Lustre: server umount lustre-MDT0000 complete [ 1182.373174] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1182.373237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1182.373275] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1182.373394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1182.373414] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1182.373429] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1182.421427] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1182.787136] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1182.787847] LustreError: 28142:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557772 with bad export cookie 11797782520124216936 [ 1182.788020] Lustre: Failing over lustre-MDT0001 [ 1182.788044] LustreError: 29095:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557772 with bad export cookie 11797782520124216936 [ 1182.788063] LustreError: 28143:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557772 with bad export cookie 11797782520124216936 [ 1182.788071] LustreError: 29095:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557772 with bad export cookie 11797782520124216936 [ 1182.788096] LustreError: 29619:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557772 with bad export cookie 11797782520124216936 [ 1182.919038] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1184.712235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1187.856268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1188.201209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1190.714046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1194.036453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1194.318530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1197.730271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1197.740420] Lustre: server umount lustre-MDT0001 complete [ 1203.388651] Lustre: lustre-MDT0000: reset Object Index mappings [ 1203.388659] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557776/real 1717557776] req@ffff880073bb6300 x1800988769773184/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557792 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1203.388691] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557776/real 1717557776] req@ffff880073bb7480 x1800988769773312/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557792 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1203.408178] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1204.380749] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1204.386080] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557777/real 1717557777] req@ffff880073bb5180 x1800988769773440/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557793 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1208.388663] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557777/real 1717557777] req@ffff880073bb5f80 x1800988769773376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557793 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1208.410801] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557781/real 1717557781] req@ffff88009d1fe300 x1800988769773568/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557797 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1208.411742] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557781/real 1717557781] req@ffff88009d1fc000 x1800988769773504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557797 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1208.411786] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff880129e48a80 x1800988769774400/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1208.411834] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff880129e48a80 x1800988769774464/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1208.524851] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff880129e48a80 x1800988769774528/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1209.530722] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1209.540701] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557782/real 1717557782] req@ffff880073bb4000 x1800988769773696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1209.573339] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557782/real 1717557782] req@ffff880073bb4e00 x1800988769773760/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1212.533405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1212.540128] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1212.540251] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1212.543865] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1212.558133] Lustre: lustre-MDT0001: reset Object Index mappings [ 1212.615657] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1212.632702] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 1213.624649] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1213.632595] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557787/real 1717557787] req@ffff880073bb5500 x1800988769773824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557803 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1213.668674] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557787/real 1717557787] req@ffff880073bb7100 x1800988769773888/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557803 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1218.636655] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1218.644276] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557791/real 1717557791] req@ffff880073bb4a80 x1800988769774016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557807 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1219.638717] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557791/real 1717557791] req@ffff880073bb7b80 x1800988769774144/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557807 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1243.916811] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557792/real 1717557792] req@ffff880129e48700 x1800988769774208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557808 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1247.326538] Lustre: DEBUG MARKER: == sanity-scrub test 9: OI scrub speed control =========== 23:23:52 (1717557832) [ 1261.339444] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 1292.281665] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1292.366367] Lustre: Failing over lustre-MDT0000 [ 1292.756818] Lustre: server umount lustre-MDT0000 complete [ 1292.758482] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1292.761602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1293.424092] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1293.424964] LustreError: 28141:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557882 with bad export cookie 11797782520124304541 [ 1293.425030] Lustre: Failing over lustre-MDT0001 [ 1293.425105] LustreError: 29950:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557882 with bad export cookie 11797782520124304541 [ 1293.425123] LustreError: 29950:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557882 with bad export cookie 11797782520124304541 [ 1293.425146] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1293.425162] LustreError: 29950:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557882 with bad export cookie 11797782520124304541 [ 1293.681478] LustreError: 28143:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717557882 with bad export cookie 11797782520124304541 [ 1295.640130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1299.048448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1299.342266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1302.387039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1305.734961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1306.009696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1309.750421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1309.760149] Lustre: server umount lustre-MDT0001 complete [ 1312.772670] Lustre: lustre-MDT0000: reset Object Index mappings [ 1312.774211] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557886/real 1717557886] req@ffff8800a1809f80 x1800988769844992/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557902 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1313.436752] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.442969] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557886/real 1717557886] req@ffff8800a1808700 x1800988769844864/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557902 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1313.780676] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.780683] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557887/real 1717557887] req@ffff8800a180ad80 x1800988769845312/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557903 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1313.780719] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557887/real 1717557887] req@ffff8800a1809880 x1800988769845184/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557903 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1313.780750] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.780759] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557886/real 1717557886] req@ffff8800a180aa00 x1800988769844928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557902 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1313.830153] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557887/real 1717557887] req@ffff8800a1808380 x1800988769845248/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557903 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1313.840071] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1314.780666] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557886/real 1717557886] req@ffff8800a180a680 x1800988769845056/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557902 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1317.840001] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557887/real 1717557887] req@ffff8800a1808000 x1800988769845120/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557903 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1318.014460] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff880129c55180 x1800988769846464/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1318.844647] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1319.019659] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557892/real 1717557892] req@ffff88009dfd7b80 x1800988769845568/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557908 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1319.022630] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557892/real 1717557892] req@ffff8800a180b480 x1800988769845376/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557908 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1319.033069] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557892/real 1717557892] req@ffff88009dfd6680 x1800988769845440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557908 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1319.035578] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1319.156915] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557892/real 1717557892] req@ffff88009dfd4a80 x1800988769845504/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557908 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1322.092650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1322.101809] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1322.196261] Lustre: lustre-MDT0001: reset Object Index mappings [ 1322.960989] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 1323.211250] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1324.213587] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1325.217619] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557897/real 1717557897] req@ffff880073a3a300 x1800988769845696/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557913 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1325.217633] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557897/real 1717557897] req@ffff880073a39880 x1800988769845888/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557913 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1325.217687] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557897/real 1717557897] req@ffff880073a3ad80 x1800988769845824/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557913 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1329.220718] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557897/real 1717557897] req@ffff880073a38a80 x1800988769845760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557913 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1330.236734] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557902/real 1717557902] req@ffff8800a2035f80 x1800988769845952/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557918 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1330.236740] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557903/real 1717557903] req@ffff8800a2036a00 x1800988769846336/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557919 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1330.236794] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557902/real 1717557902] req@ffff8800a2034a80 x1800988769846144/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557918 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1362.185291] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557902/real 1717557902] req@ffff8800a2034000 x1800988769846016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557918 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1365.950349] Lustre: DEBUG MARKER: == sanity-scrub test 10a: non-stopped OI scrub should auto restarts after MDS remount (1) ========================================================== 23:25:50 (1717557950) [ 1380.349795] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 1424.580664] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1424.660612] Lustre: Failing over lustre-MDT0000 [ 1426.050268] Lustre: server umount lustre-MDT0000 complete [ 1426.051721] LustreError: 29950:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558015 with bad export cookie 11797782520124514310 [ 1426.051733] LustreError: 28142:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558015 with bad export cookie 11797782520124514310 [ 1426.051772] LustreError: 28141:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558015 with bad export cookie 11797782520124514310 [ 1426.051798] LustreError: 28143:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558015 with bad export cookie 11797782520124514310 [ 1426.051819] LustreError: 28142:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558015 with bad export cookie 11797782520124514310 [ 1426.051907] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1426.280414] Lustre: Failing over lustre-MDT0001 [ 1428.290822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1431.742538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1432.083553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1434.768170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1437.107356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1437.356301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1440.196975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1440.204546] Lustre: server umount lustre-MDT0001 complete [ 1445.051642] Lustre: lustre-MDT0000: reset Object Index mappings [ 1445.053217] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558018/real 1717558018] req@ffff88009aa61c00 x1800988769936512/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558034 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1445.060588] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.060606] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558018/real 1717558018] req@ffff88009aa60380 x1800988769936448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558034 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1445.060618] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.060651] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558018/real 1717558018] req@ffff88009aa61500 x1800988769936384/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558034 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1445.097825] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.101774] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558018/real 1717558018] req@ffff88009aa62680 x1800988769936576/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558034 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1447.064695] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1447.097654] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558020/real 1717558020] req@ffff88009aa63480 x1800988769936768/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558036 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1447.110612] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558020/real 1717558020] req@ffff88009aa60e00 x1800988769936704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558036 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1447.110653] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558020/real 1717558020] req@ffff88009aa60a80 x1800988769936640/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558036 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1451.110968] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558020/real 1717558020] req@ffff88009aa63800 x1800988769936832/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558036 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1451.244749] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88012b052a00 x1800988769937792/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1452.142615] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1452.150459] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558025/real 1717558025] req@ffff88006fa8d880 x1800988769937024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558041 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1452.249600] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558025/real 1717558025] req@ffff88006fa8e300 x1800988769936960/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558041 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1452.266129] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558025/real 1717558025] req@ffff88006fa8d500 x1800988769937088/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558041 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1452.363449] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558025/real 1717558025] req@ffff88006fa8e680 x1800988769937152/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558041 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1454.849096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1454.859466] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1454.961213] Lustre: lustre-MDT0001: reset Object Index mappings [ 1455.977654] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 1456.012469] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1456.980635] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1457.260557] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558030/real 1717558030] req@ffff88006fa8ea00 x1800988769937216/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558046 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1457.260561] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558030/real 1717558030] req@ffff88006fa8ed80 x1800988769937280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558046 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1457.260582] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558030/real 1717558030] req@ffff88006fa8f480 x1800988769937408/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558046 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1458.679711] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558030/real 1717558030] req@ffff88006fa8f100 x1800988769937344/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558046 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1460.735666] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200004282:0x1:0x0]/32041: rc = 0 [ 1462.268612] Lustre: lustre-MDT0001: trigger OI scrub by RPC for [0x240004281:0x1:0x0]/32006 with flags 0x52: rc = 0 [ 1463.152763] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558034/real 1717558034] req@ffff88012b053800 x1800988769937664/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558050 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1463.224021] Lustre: Failing over lustre-MDT0000 [ 1464.513463] Lustre: server umount lustre-MDT0000 complete [ 1464.514929] LustreError: 29095:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558053 with bad export cookie 11797782520125215171 [ 1464.515080] Lustre: Failing over lustre-MDT0001 [ 1464.515147] LustreError: 28143:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558053 with bad export cookie 11797782520125215171 [ 1464.515215] LustreError: 28143:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558053 with bad export cookie 11797782520125215171 [ 1464.515348] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1464.631082] LustreError: 29619:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558053 with bad export cookie 11797782520125215171 [ 1467.509836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1474.514883] Lustre: server umount lustre-MDT0001 complete [ 1474.619806] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88012e56bb80 x1800988769950016/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1474.631279] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1474.631298] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1474.631330] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1474.631338] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1474.631433] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1474.631464] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1474.631501] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1475.403259] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1477.516148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1477.578159] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1478.332387] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 1479.726381] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1479.731827] Lustre: Failing over lustre-MDT0000 [ 1479.733942] LustreError: 23365:0:(/home/green/git/lustre-release/lustre/osp/osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1479.738509] LustreError: 23365:0:(/home/green/git/lustre-release/lustre/lod/lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1479.744891] LustreError: 23365:0:(/home/green/git/lustre-release/lustre/lod/lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 0, failed: rc = -5 [ 1479.744945] Lustre: 23366:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1479.752466] LustreError: 24853:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2927:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1479.756213] Lustre: 23366:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1479.766132] Lustre: 23366:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1479.769971] LustreError: 23366:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1281:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88012b768700 x1800988769957888/t0(0) o1000->lustre-MDT0001-osp-MDT0000@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'tgt_recover_0.0' uid:0 gid:0 [ 1479.776697] LustreError: 23366:0:(/home/green/git/lustre-release/lustre/osp/osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1479.792358] LustreError: 23366:0:(/home/green/git/lustre-release/lustre/lod/lod_dev.c:365:lod_sub_recreate_llog()) lustre-MDT0000-mdtlov: can't access update_log: rc = -5 [ 1479.844972] Lustre: 23366:0:(/home/green/git/lustre-release/lustre/mdt/mdt_handler.c:7962:mdt_postrecov()) lustre-MDT0000: auto trigger paused LFSCK failed: rc = -6 [ 1480.784998] Lustre: server umount lustre-MDT0000 complete [ 1480.785805] LustreError: 28143:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558070 with bad export cookie 11797782520125216928 [ 1480.785940] Lustre: Failing over lustre-MDT0001 [ 1480.785954] LustreError: 29095:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558070 with bad export cookie 11797782520125216928 [ 1480.785964] LustreError: 28142:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558070 with bad export cookie 11797782520125216928 [ 1480.785976] LustreError: 29095:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558070 with bad export cookie 11797782520125216928 [ 1480.785997] LustreError: 29619:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558070 with bad export cookie 11797782520125216928 [ 1480.786153] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1480.786175] LustreError: 24223:0:(/home/green/git/lustre-release/lustre/osp/osp_object.c:637:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000009:0x0:0x0]: rc = -5 [ 1480.786179] LustreError: 24223:0:(/home/green/git/lustre-release/lustre/lod/lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't get id from catalogs: rc = -5 [ 1480.898118] LustreError: 24223:0:(/home/green/git/lustre-release/lustre/lod/lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0000-osp-MDT0001: get update log duration 3, retries 0, failed: rc = -5 [ 1481.725643] Lustre: server umount lustre-MDT0001 complete [ 1481.784579] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558055/real 1717558055] req@ffff88012e568e00 x1800988769949440/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558071 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1481.788567] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558055/real 1717558055] req@ffff88012e56b100 x1800988769949312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558071 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1481.791846] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558055/real 1717558055] req@ffff88012e56a680 x1800988769949248/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558071 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1483.007563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1484.596632] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558055/real 1717558055] req@ffff88012e569500 x1800988769949376/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558071 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1484.596634] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558058/real 1717558058] req@ffff88012e56b800 x1800988769949696/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558074 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1484.787663] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558058/real 1717558058] req@ffff88012e568380 x1800988769949632/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558074 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1484.805584] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558058/real 1717558058] req@ffff88012e569c00 x1800988769949504/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558074 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1489.604597] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558058/real 1717558058] req@ffff88012e56ad80 x1800988769949568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558074 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1489.606592] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558063/real 1717558063] req@ffff88012e56a300 x1800988769949888/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558079 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1489.606595] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558063/real 1717558063] req@ffff88012e569880 x1800988769949824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558079 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1489.612571] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558063/real 1717558063] req@ffff88012e569180 x1800988769949760/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558079 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1490.596814] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558063/real 1717558063] req@ffff88012e568000 x1800988769949952/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558079 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1490.682191] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88012e56b800 x1800988769959040/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1490.682215] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1490.682249] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1491.430300] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1493.616297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1493.689546] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1494.457966] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 1497.703041] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1498.708591] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1499.263625] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558072/real 1717558072] req@ffff88006fa8c380 x1800988769958656/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558088 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1499.708620] Lustre: DEBUG MARKER: == sanity-scrub test 11: OI scrub skips the new created objects only once ========================================================== 23:28:07 (1717558087) [ 1501.805122] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558072/real 1717558072] req@ffff88006fa8ed80 x1800988769958528/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558088 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1502.708567] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 1503.703589] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558076/real 1717558076] req@ffff88006fa8e300 x1800988769958784/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558092 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1503.740573] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558077/real 1717558077] req@ffff88012e56ad80 x1800988769958912/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558093 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1504.716572] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558077/real 1717558077] req@ffff88012e569c00 x1800988769958976/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558093 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1506.868597] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558076/real 1717558076] req@ffff88006fa8f800 x1800988769958848/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558092 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1506.868601] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558080/real 1717558080] req@ffff88012e56b100 x1800988769959168/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558096 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1511.975947] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558080/real 1717558080] req@ffff88012e568380 x1800988769959104/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558096 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1523.630351] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1525.322644] Lustre: DEBUG MARKER: == sanity-scrub test 12: OI scrub can rebuild invalid /O entries ========================================================== 23:28:32 (1717558112) [ 1525.344816] Lustre: Failing over lustre-OST0000 [ 1527.669217] Lustre: server umount lustre-OST0000 complete [ 1527.670599] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1527.673447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1527.706025] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1527.709647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to completeLustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1528.872504] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck allLustre: lustre-OST0000: Denying connection for new client 88153284-c06c-40f1-83d9-cb2f9bc4ecb3 (at 192.168.202.42@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 1712.971229] Lustre: DEBUG MARKER: == sanity-scrub test 13: OI scrub can rebuild missed /O entries ========================================================== 23:31:41 (1717558301)Lustre: Failing over lustre-OST0000 [ 1714.738133] Lustre: server umount lustre-OST0000 completeLustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1714.984024] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to completeLustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1714.985292] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1714.995381] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1716.821209] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1716.826769] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1716.830253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1718.043892] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1718.298044] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnectLustre: DEBUG MARKER: == sanity-scrub test 14: OI scrub can repair OST objects under lost+found ========================================================== 23:34:56 (1717558496) [ 1913.911469] Lustre: Failing over lustre-OST0000Lustre: server umount lustre-OST0000 complete [ 1916.343992] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1916.344075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1916.344087] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1916.344129] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1916.344166] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1916.349450] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1916.384309] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1917.189089] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1917.189217] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1917.189402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1917.923374] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1917.931463] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1917.937582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1919.839429] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1920.019320] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnectLustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete[ 1923.029167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to completeLustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to completeLustre: server umount lustre-MDT0000 complete [ 1929.869274] LustreError: 29619:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558519 with bad export cookie 11797782520125218076LustreError: 28143:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558519 with bad export cookie 11797782520125218076 [ 1929.870750] LustreError: 29095:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558519 with bad export cookie 11797782520125218076 [ 1929.870753] LustreError: 28143:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558519 with bad export cookie 11797782520125218076 [ 1929.870810] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1929.997975] LustreError: 28143:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558519 with bad export cookie 11797782520125218076 [ 1941.475152] Lustre: server umount lustre-MDT0001 complete [ 1952.868037] Lustre: server umount lustre-OST0000 complete [ 1956.449495] Lustre: server umount lustre-OST0001 complete [ 1961.844461] Lustre: DEBUG MARKER: == sanity-scrub test 15: Dryrun mode OI scrub ============ 23:35:44 (1717558544) [ 1963.519459] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 1966.813081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1969.359993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1971.490766] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 1971.493010] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1971.495828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1973.556976] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1973.560131] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1977.385156] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local[ 1980.401022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1980.429431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1980.603601] Lustre: lustre-MDT0000: new disk, initializingLustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all[ 1985.309461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1985.324645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1985.383310] Lustre: lustre-MDT0001: new disk, initializingLustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1989.800665] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1989.803649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1989.818481] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1989.821431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1989.889191] Lustre: lustre-OST0000: new disk, initializing [ 1991.069407] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck allLDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1994.375270] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1994.393582] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1994.395983] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1994.416057] Lustre: lustre-OST0001: new disk, initializing [ 1995.594838] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck allLustre: DEBUG MARKER: Using TIMEOUT=20 [ 2010.793058] Lustre: Failing over lustre-MDT0000Lustre: server umount lustre-MDT0000 complete [ 2011.777964] LustreError: 11539:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558601 with bad export cookie 11797782520125338014Lustre: Failing over lustre-MDT0001 [ 2011.778916] LustreError: 11541:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558601 with bad export cookie 11797782520125338014 [ 2011.778951] LustreError: 14551:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558601 with bad export cookie 11797782520125338014 [ 2011.778953] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2011.778957] LustreError: 11541:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558601 with bad export cookie 11797782520125338014 [ 2011.886281] LustreError: 14551:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558601 with bad export cookie 11797782520125338014 [ 2013.251095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2015.791406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2016.068700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2018.480751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2021.899472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2022.219863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2026.198064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2026.205967] Lustre: server umount lustre-MDT0001 complete [ 2031.716651] Lustre: lustre-MDT0000: reset Object Index mappings [ 2031.719989] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558605/real 1717558605] req@ffff88009def8e00 x1800988770175744/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558621 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2032.736658] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2032.736694] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558606/real 1717558606] req@ffff88009def8380 x1800988770176064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2032.736726] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558606/real 1717558606] req@ffff88009def8700 x1800988770176192/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2032.736735] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558606/real 1717558606] req@ffff88009def9180 x1800988770176128/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2032.736762] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2032.736771] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2032.736834] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558605/real 1717558605] req@ffff88009defbb80 x1800988770175936/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558621 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2032.808489] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558606/real 1717558606] req@ffff88009def8000 x1800988770176000/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2032.816664] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2032.820796] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558605/real 1717558605] req@ffff88009defaa00 x1800988770175872/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558621 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2036.816894] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558605/real 1717558605] req@ffff88009defb480 x1800988770175808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558621 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2036.976764] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff880130c4dc00 x1800988770177344/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 2037.736681] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2037.996586] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558611/real 1717558611] req@ffff88009defad80 x1800988770176448/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2037.998600] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558611/real 1717558611] req@ffff88009def8a80 x1800988770176256/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2037.998635] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558611/real 1717558611] req@ffff88009defb800 x1800988770176384/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2038.012338] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558611/real 1717558611] req@ffff88009defb100 x1800988770176320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2038.125476] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2040.985605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2040.994796] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2041.109922] Lustre: lustre-MDT0001: reset Object Index mappings [ 2042.127946] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 2042.139812] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2043.125612] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2044.132648] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558616/real 1717558616] req@ffff88009def9880 x1800988770176640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558632 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2044.132728] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558616/real 1717558616] req@ffff88012e56ad80 x1800988770176768/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558632 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2044.166667] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558616/real 1717558616] req@ffff88012e569500 x1800988770176704/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558632 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2048.132639] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558616/real 1717558616] req@ffff88009def9500 x1800988770176576/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558632 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2048.150637] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558621/real 1717558621] req@ffff88012e5b8380 x1800988770176960/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2049.150684] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558621/real 1717558621] req@ffff88012e5b8a80 x1800988770176896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2049.165577] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558621/real 1717558621] req@ffff88012e5b8000 x1800988770177024/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2058.319402] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558622/real 1717558622] req@ffff88012e5ba300 x1800988770177088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558638 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2062.031891] Lustre: DEBUG MARKER: == sanity-scrub test 16: Initial OI scrub can rebuild crashed index objects ========================================================== 23:37:26 (1717558646) [ 2076.017143] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 2091.281936] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2091.286758] Lustre: Failing over lustre-MDT0000 [ 2091.288232] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000001:0x3:0x0]: rc = 0 [ 2091.293658] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2:0x0]: rc = 0 [ 2091.298489] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x9:0x0]: rc = 0 [ 2091.302836] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xa:0x0]: rc = 0 [ 2091.306585] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xb:0x0]: rc = 0 [ 2091.310760] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xc:0x0]: rc = 0 [ 2091.314588] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xd:0x0]: rc = 0 [ 2091.318460] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0xe:0x0]: rc = 0 [ 2091.322113] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x13:0x0]: rc = 0 [ 2091.325872] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x14:0x0]: rc = 0 [ 2091.329387] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x15:0x0]: rc = 0 [ 2091.332873] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x16:0x0]: rc = 0 [ 2091.336308] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x17:0x0]: rc = 0 [ 2091.339689] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x18:0x0]: rc = 0 [ 2091.343081] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x19:0x0]: rc = 0 [ 2091.346314] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1a:0x0]: rc = 0 [ 2091.349595] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1b:0x0]: rc = 0 [ 2091.353070] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1c:0x0]: rc = 0 [ 2091.356461] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1d:0x0]: rc = 0 [ 2091.359776] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1e:0x0]: rc = 0 [ 2091.363076] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x1f:0x0]: rc = 0 [ 2091.366359] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x20:0x0]: rc = 0 [ 2091.369629] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x21:0x0]: rc = 0 [ 2091.373007] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x22:0x0]: rc = 0 [ 2091.376307] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x23:0x0]: rc = 0 [ 2091.379652] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x25:0x0]: rc = 0 [ 2091.383436] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x26:0x0]: rc = 0 [ 2091.387245] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x27:0x0]: rc = 0 [ 2091.391141] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x28:0x0]: rc = 0 [ 2091.394935] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x29:0x0]: rc = 0 [ 2091.398837] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2a:0x0]: rc = 0 [ 2091.402582] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2b:0x0]: rc = 0 [ 2091.406435] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2c:0x0]: rc = 0 [ 2091.410321] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2d:0x0]: rc = 0 [ 2091.414032] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2e:0x0]: rc = 0 [ 2091.417777] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x2f:0x0]: rc = 0 [ 2091.421278] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x30:0x0]: rc = 0 [ 2091.424751] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x31:0x0]: rc = 0 [ 2091.427937] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x32:0x0]: rc = 0 [ 2091.431339] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x33:0x0]: rc = 0 [ 2091.434965] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x34:0x0]: rc = 0 [ 2091.438805] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000003:0x35:0x0]: rc = 0 [ 2091.442445] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x1:0x0]: rc = 0 [ 2091.446031] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x2:0x0]: rc = 0 [ 2091.449648] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x3:0x0]: rc = 0 [ 2091.453340] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x20001:0x0]: rc = 0 [ 2091.457053] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x20002:0x0]: rc = 0 [ 2091.460644] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000005:0x20003:0x0]: rc = 0 [ 2091.464071] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x10000:0x0]: rc = 0 [ 2091.467430] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x20000:0x0]: rc = 0 [ 2091.471076] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x1010000:0x0]: rc = 0 [ 2091.474595] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x1020000:0x0]: rc = 0 [ 2091.478197] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x2010000:0x0]: rc = 0 [ 2091.534011] Lustre: 26955:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0000: backup index [0x200000006:0x2020000:0x0]: rc = 0 [ 2092.213133] Lustre: server umount lustre-MDT0000 complete [ 2092.217310] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2092.224843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2092.725273] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2092.726774] LustreError: 14551:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558682 with bad export cookie 11797782520125354695 [ 2092.727007] Lustre: Failing over lustre-MDT0001 [ 2092.727011] LustreError: 11540:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558682 with bad export cookie 11797782520125354695 [ 2092.727055] LustreError: 11539:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558682 with bad export cookie 11797782520125354695 [ 2092.727059] LustreError: 11540:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558682 with bad export cookie 11797782520125354695 [ 2092.727116] LustreError: 11539:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558682 with bad export cookie 11797782520125354695 [ 2092.782456] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2092.789715] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000001:0x3:0x0]: rc = 0 [ 2092.798386] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x3:0x0]: rc = 0 [ 2092.807013] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x4:0x0]: rc = 0 [ 2092.813965] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x5:0x0]: rc = 0 [ 2092.821498] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x6:0x0]: rc = 0 [ 2092.828377] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x7:0x0]: rc = 0 [ 2092.834279] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x8:0x0]: rc = 0 [ 2092.839207] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0xd:0x0]: rc = 0 [ 2092.844399] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0xe:0x0]: rc = 0 [ 2092.849457] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0xf:0x0]: rc = 0 [ 2092.854194] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x10:0x0]: rc = 0 [ 2092.858509] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x11:0x0]: rc = 0 [ 2092.862750] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x12:0x0]: rc = 0 [ 2092.867136] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x13:0x0]: rc = 0 [ 2092.870948] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x14:0x0]: rc = 0 [ 2092.875196] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x15:0x0]: rc = 0 [ 2092.878840] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x16:0x0]: rc = 0 [ 2092.882851] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x17:0x0]: rc = 0 [ 2092.886650] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x18:0x0]: rc = 0 [ 2092.890622] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x19:0x0]: rc = 0 [ 2092.894048] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1a:0x0]: rc = 0 [ 2092.897660] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1b:0x0]: rc = 0 [ 2092.901999] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1c:0x0]: rc = 0 [ 2092.907128] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1d:0x0]: rc = 0 [ 2092.912429] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1f:0x0]: rc = 0 [ 2092.915993] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x20:0x0]: rc = 0 [ 2092.920580] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x21:0x0]: rc = 0 [ 2092.925887] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x22:0x0]: rc = 0 [ 2092.930193] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x23:0x0]: rc = 0 [ 2092.933629] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x24:0x0]: rc = 0 [ 2092.936853] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x25:0x0]: rc = 0 [ 2092.940126] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x26:0x0]: rc = 0 [ 2092.943597] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x27:0x0]: rc = 0 [ 2092.946811] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x28:0x0]: rc = 0 [ 2092.950399] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x29:0x0]: rc = 0 [ 2092.955850] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2a:0x0]: rc = 0 [ 2092.959384] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2b:0x0]: rc = 0 [ 2092.962941] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2c:0x0]: rc = 0 [ 2092.968254] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2d:0x0]: rc = 0 [ 2092.971628] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2e:0x0]: rc = 0 [ 2092.974984] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2f:0x0]: rc = 0 [ 2092.978062] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x1:0x0]: rc = 0 [ 2092.981379] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x2:0x0]: rc = 0 [ 2092.984643] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x3:0x0]: rc = 0 [ 2092.988171] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20001:0x0]: rc = 0 [ 2092.991805] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20002:0x0]: rc = 0 [ 2093.106980] Lustre: 27221:0:(/home/green/git/lustre-release/lustre/obdclass/scrub.c:1101:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20003:0x0]: rc = 0 [ 2096.173969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2096.200918] Lustre: server umount lustre-MDT0001 complete [ 2096.203408] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace' with [0x200000003:0x13:0x0]: rc = 0 [ 2096.209750] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 2096.217450] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xb:0x0]: rc = 0 [ 2096.222485] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000003:0xe:0x0]: rc = 0 [ 2096.227301] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 2096.231901] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 2096.238118] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0x9:0x0]: rc = 0 [ 2096.243410] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x2020000-MDT0000' with [0x200000005:0x20003:0x0]: rc = 0 [ 2096.249000] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 2096.256219] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x20000-MDT0000' with [0x200000005:0x20001:0x0]: rc = 0 [ 2096.261588] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xa:0x0]: rc = 0 [ 2096.266310] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000003:0xd:0x0]: rc = 0 [ 2096.270784] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000003:0xc:0x0]: rc = 0 [ 2096.275368] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x1020000-MDT0000' with [0x200000005:0x20002:0x0]: rc = 0 [ 2096.280291] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x2:0x0]: rc = 0 [ 2096.285391] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_06' with [0x200000003:0x2b:0x0]: rc = 0 [ 2096.291361] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_02' with [0x200000003:0x16:0x0]: rc = 0 [ 2096.296728] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_14' with [0x200000003:0x22:0x0]: rc = 0 [ 2096.302732] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_05' with [0x200000003:0x19:0x0]: rc = 0 [ 2096.307795] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_07' with [0x200000003:0x1b:0x0]: rc = 0 [ 2096.313472] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_06' with [0x200000003:0x1a:0x0]: rc = 0 [ 2096.318644] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_10' with [0x200000003:0x2f:0x0]: rc = 0 [ 2096.323096] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_01' with [0x200000003:0x15:0x0]: rc = 0 [ 2096.327757] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_05' with [0x200000003:0x2a:0x0]: rc = 0 [ 2096.332389] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_10' with [0x200000003:0x1e:0x0]: rc = 0 [ 2096.337463] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_12' with [0x200000003:0x20:0x0]: rc = 0 [ 2096.342444] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_14' with [0x200000003:0x33:0x0]: rc = 0 [ 2096.347005] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_03' with [0x200000003:0x28:0x0]: rc = 0 [ 2096.351644] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_00' with [0x200000003:0x14:0x0]: rc = 0 [ 2096.356437] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_07' with [0x200000003:0x2c:0x0]: rc = 0 [ 2096.361128] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_15' with [0x200000003:0x34:0x0]: rc = 0 [ 2096.366061] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_09' with [0x200000003:0x2e:0x0]: rc = 0 [ 2096.370811] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_13' with [0x200000003:0x21:0x0]: rc = 0 [ 2096.375737] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_00' with [0x200000003:0x25:0x0]: rc = 0 [ 2096.380843] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_01' with [0x200000003:0x26:0x0]: rc = 0 [ 2096.385276] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_02' with [0x200000003:0x27:0x0]: rc = 0 [ 2096.391349] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_04' with [0x200000003:0x18:0x0]: rc = 0 [ 2096.396358] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_13' with [0x200000003:0x32:0x0]: rc = 0 [ 2096.401424] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_03' with [0x200000003:0x17:0x0]: rc = 0 [ 2096.405682] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_08' with [0x200000003:0x1c:0x0]: rc = 0 [ 2096.409848] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_08' with [0x200000003:0x2d:0x0]: rc = 0 [ 2096.414837] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_09' with [0x200000003:0x1d:0x0]: rc = 0 [ 2096.420112] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_04' with [0x200000003:0x29:0x0]: rc = 0 [ 2096.424768] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_11' with [0x200000003:0x1f:0x0]: rc = 0 [ 2096.429885] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_11' with [0x200000003:0x30:0x0]: rc = 0 [ 2096.434783] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_layout_12' with [0x200000003:0x31:0x0]: rc = 0 [ 2096.439461] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index 'lfsck_namespace_15' with [0x200000003:0x23:0x0]: rc = 0 [ 2096.444301] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 2096.448798] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 2096.453409] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 2096.457196] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 2096.460930] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 2102.727148] Lustre: 27879:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 2102.859267] Lustre: Evicted from MGS (at 192.168.202.142@tcp) after server handle changed from 0x0 to 0xa3ba24b1d962e7ac [ 2103.977658] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2106.668235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2106.693384] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2106.698912] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace' with [0x200000003:0xd:0x0]: rc = 0 [ 2106.705742] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 2106.712845] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_06' with [0x200000003:0x25:0x0]: rc = 0 [ 2106.719739] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_09' with [0x200000003:0x17:0x0]: rc = 0 [ 2106.726887] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_10' with [0x200000003:0x29:0x0]: rc = 0 [ 2106.731402] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_08' with [0x200000003:0x16:0x0]: rc = 0 [ 2106.735407] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_12' with [0x200000003:0x2b:0x0]: rc = 0 [ 2106.739940] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_04' with [0x200000003:0x12:0x0]: rc = 0 [ 2106.747984] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_13' with [0x200000003:0x1b:0x0]: rc = 0 [ 2106.755374] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_06' with [0x200000003:0x14:0x0]: rc = 0 [ 2106.760818] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_14' with [0x200000003:0x1c:0x0]: rc = 0 [ 2106.766296] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_03' with [0x200000003:0x22:0x0]: rc = 0 [ 2106.771056] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_01' with [0x200000003:0xf:0x0]: rc = 0 [ 2106.774858] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_08' with [0x200000003:0x27:0x0]: rc = 0 [ 2106.779309] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_05' with [0x200000003:0x13:0x0]: rc = 0 [ 2106.783364] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_05' with [0x200000003:0x24:0x0]: rc = 0 [ 2106.787614] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_00' with [0x200000003:0x1f:0x0]: rc = 0 [ 2106.791454] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_02' with [0x200000003:0x10:0x0]: rc = 0 [ 2106.795552] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_13' with [0x200000003:0x2c:0x0]: rc = 0 [ 2106.799670] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_11' with [0x200000003:0x2a:0x0]: rc = 0 [ 2106.803844] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_00' with [0x200000003:0xe:0x0]: rc = 0 [ 2106.807868] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_03' with [0x200000003:0x11:0x0]: rc = 0 [ 2106.812382] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_02' with [0x200000003:0x21:0x0]: rc = 0 [ 2106.816274] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_15' with [0x200000003:0x1d:0x0]: rc = 0 [ 2106.820165] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_09' with [0x200000003:0x28:0x0]: rc = 0 [ 2106.825001] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_01' with [0x200000003:0x20:0x0]: rc = 0 [ 2106.829065] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_11' with [0x200000003:0x19:0x0]: rc = 0 [ 2106.832997] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_07' with [0x200000003:0x15:0x0]: rc = 0 [ 2106.837952] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_07' with [0x200000003:0x26:0x0]: rc = 0 [ 2106.842553] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_15' with [0x200000003:0x2e:0x0]: rc = 0 [ 2106.847374] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_14' with [0x200000003:0x2d:0x0]: rc = 0 [ 2106.851846] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_12' with [0x200000003:0x1a:0x0]: rc = 0 [ 2106.856506] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_10' with [0x200000003:0x18:0x0]: rc = 0 [ 2106.860987] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_04' with [0x200000003:0x23:0x0]: rc = 0 [ 2106.865108] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x2020000' with [0x200000003:0x8:0x0]: rc = 0 [ 2106.869921] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 2106.873465] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x20000-MDT0001' with [0x200000005:0x20001:0x0]: rc = 0 [ 2106.878630] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x4:0x0]: rc = 0 [ 2106.882905] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x1020000-MDT0001' with [0x200000005:0x20002:0x0]: rc = 0 [ 2106.887428] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x3:0x0]: rc = 0 [ 2106.891208] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x2020000-MDT0001' with [0x200000005:0x20003:0x0]: rc = 0 [ 2106.897034] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 2106.901180] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 2106.905413] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x1020000' with [0x200000003:0x7:0x0]: rc = 0 [ 2106.909681] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x5:0x0]: rc = 0 [ 2106.964401] Lustre: 28763:0:(/home/green/git/lustre-release/lustre/osd-ldiskfs/osd_scrub.c:1818:osd_index_restore()) lustre-MDT0001: restore index '0x20000' with [0x200000003:0x6:0x0]: rc = 0 [ 2106.964444] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 2106.964473] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2106.964599] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2106.964625] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2107.764086] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2108.397005] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2108.404202] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2108.871795] Lustre: lustre-MDT0000: Denying connection for new client 99b72272-2b9b-4ea3-9f52-e6998d5d7297 (at 192.168.202.42@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2108.883518] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558682/real 1717558682] req@ffff8800722a0a80 x1800988770236096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558698 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2108.985753] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558682/real 1717558682] req@ffff8800722a2300 x1800988770236160/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558698 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2109.000649] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558682/real 1717558682] req@ffff8800722a1180 x1800988770236032/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558698 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2113.988638] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558682/real 1717558682] req@ffff8800722a0700 x1800988770236224/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558698 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2114.007260] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558687/real 1717558687] req@ffff88009aa60380 x1800988770236480/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558703 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2114.015659] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558687/real 1717558687] req@ffff88009cf23100 x1800988770236352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558703 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2114.044600] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558687/real 1717558687] req@ffff88009cf23b80 x1800988770236288/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558703 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2115.679347] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558687/real 1717558687] req@ffff88009cf23480 x1800988770236416/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558703 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2116.526255] Lustre: DEBUG MARKER: == sanity-scrub test 17a: ENOSPC on OI insert shouldn't leak inodes ========================================================== 23:38:24 (1717558704) [ 2116.581515] Lustre: Failing over lustre-MDT0000 [ 2117.012913] Lustre: server umount lustre-MDT0000 complete [ 2117.012922] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2117.012982] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2117.013040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2117.013099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2117.013114] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2117.013149] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2117.062601] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2118.420022] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.012561] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.012568] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558692/real 1717558692] req@ffff8800722a3100 x1800988770236800/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558708 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2119.025430] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558692/real 1717558692] req@ffff8800722a2a00 x1800988770236736/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558708 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2119.062585] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558692/real 1717558692] req@ffff8800722a0e00 x1800988770236608/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558708 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2120.687466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2120.723422] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558692/real 1717558692] req@ffff8800722a2d80 x1800988770236672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558708 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2121.754653] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2122.596184] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2123.422937] Lustre: DEBUG MARKER: == sanity-scrub test 17b: ENOSPC on .. insertion shouldn't leak inodes ========================================================== 23:38:31 (1717558711) [ 2126.267883] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2126.317422] Lustre: Failing over lustre-MDT0000 [ 2128.434365] Lustre: server umount lustre-MDT0000 complete [ 2130.109144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.135577] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2130.181090] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2130.181096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.181122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.191966] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.935338] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.841969] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2133.439642] Lustre: DEBUG MARKER: == sanity-scrub test 18: test mount -o resetoi to recreate OI files ========================================================== 23:38:40 (1717558720) [ 2141.405141] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2141.463666] Lustre: Failing over lustre-MDT0000 [ 2142.392906] Lustre: server umount lustre-MDT0000 complete [ 2142.393830] LustreError: 11540:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558731 with bad export cookie 11797782520125387378 [ 2142.393898] Lustre: Failing over lustre-MDT0001 [ 2142.393902] LustreError: 11539:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558731 with bad export cookie 11797782520125387378 [ 2142.393931] LustreError: 26545:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558731 with bad export cookie 11797782520125387378 [ 2142.393942] LustreError: 11539:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558731 with bad export cookie 11797782520125387378 [ 2142.393957] LustreError: 26545:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558731 with bad export cookie 11797782520125387378 [ 2142.525638] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2144.627606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2152.393995] Lustre: server umount lustre-MDT0001 complete [ 2152.515804] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8800a73e6300 x1800988770302592/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 2153.284542] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2155.500702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2155.566194] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2155.566202] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2155.566234] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 2155.566266] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2155.566409] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2156.451095] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2157.255431] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2157.261544] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2160.565654] Lustre: lustre-MDT0000: Denying connection for new client 1fac5c4f-4c5e-4895-adc9-a008cf208ee4 (at 192.168.202.42@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2160.585576] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558734/real 1717558734] req@ffff8800a73e4700 x1800988770301952/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558750 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2160.595574] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558734/real 1717558734] req@ffff8800a73e7800 x1800988770302016/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558750 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2160.597254] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558734/real 1717558734] req@ffff8800a73e7480 x1800988770301888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558750 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2162.595589] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558734/real 1717558734] req@ffff8800a73e4000 x1800988770301824/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558750 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2162.609592] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558736/real 1717558736] req@ffff8800a73e4e00 x1800988770302272/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2162.619877] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558736/real 1717558736] req@ffff8800a73e4a80 x1800988770302144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2162.627585] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558736/real 1717558736] req@ffff8800a73e7b80 x1800988770302080/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2162.943893] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558736/real 1717558736] req@ffff8800a73e4380 x1800988770302208/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2163.011175] Lustre: Failing over lustre-MDT0000 [ 2163.953030] Lustre: server umount lustre-MDT0000 complete [ 2163.954201] LustreError: 12491:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558753 with bad export cookie 11797782520125414594 [ 2163.954205] LustreError: 11540:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558753 with bad export cookie 11797782520125414594 [ 2163.954234] LustreError: 11541:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558753 with bad export cookie 11797782520125414594 [ 2163.954257] LustreError: 11541:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558753 with bad export cookie 11797782520125414594 [ 2163.954267] Lustre: Failing over lustre-MDT0001 [ 2164.067818] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2166.580792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2166.587009] Lustre: server umount lustre-MDT0001 complete [ 2167.604690] Lustre: lustre-MDT0000: reset Object Index mappings [ 2167.604706] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558741/real 1717558741] req@ffff8800a73e5500 x1800988770302464/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558757 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2167.604723] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558741/real 1717558741] req@ffff8800a73e5880 x1800988770302400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558757 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2167.604793] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558741/real 1717558741] req@ffff8800a73e5f80 x1800988770302336/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558757 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2173.953935] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558741/real 1717558741] req@ffff8800a73e6680 x1800988770302528/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558757 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2174.090799] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88008d548e00 x1800988770318912/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 2174.949258] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2177.616535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2177.687089] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2177.687169] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 2177.687189] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.687203] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.687246] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2178.472311] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2179.260247] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2179.263366] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2181.686714] Lustre: lustre-MDT0000: Denying connection for new client b610c18b-c2c2-4adb-91d2-74b4f1c621fd (at 192.168.202.42@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2181.703647] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558755/real 1717558755] req@ffff88012c1f8a80 x1800988770318592/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558771 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2181.714695] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558755/real 1717558755] req@ffff88012c1f8000 x1800988770318400/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558771 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2181.718934] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558755/real 1717558755] req@ffff88012c1f9880 x1800988770318464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558771 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2185.724581] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558755/real 1717558755] req@ffff88012c1f9c00 x1800988770318528/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558771 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2185.734586] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558759/real 1717558759] req@ffff88012c1fb800 x1800988770318656/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558775 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2185.734619] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558759/real 1717558759] req@ffff88012c1f8e00 x1800988770318784/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558775 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2185.762583] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558759/real 1717558759] req@ffff88012c1f8380 x1800988770318848/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558775 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2185.818945] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558759/real 1717558759] req@ffff88012c1faa00 x1800988770318720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558775 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2187.732844] Lustre: DEBUG MARKER: == sanity-scrub test 19: LFSCK can fix multiple linked files on OST ========================================================== 23:39:34 (1717558774) [ 2187.732900] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2187.736739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2187.736789] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.732664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.732680] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558763/real 1717558763] req@ffff88008d549500 x1800988770318976/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558779 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2189.732708] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558763/real 1717558763] req@ffff88008d54a300 x1800988770319104/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558779 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2189.752338] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558763/real 1717558763] req@ffff88008d54bb80 x1800988770319168/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558779 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2193.041455] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558763/real 1717558763] req@ffff88008d54ad80 x1800988770319040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558779 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2193.985998] Lustre: server umount lustre-MDT0000 complete [ 2193.986741] LustreError: 11540:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558783 with bad export cookie 11797782520125416232 [ 2193.986745] LustreError: 11541:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558783 with bad export cookie 11797782520125416232 [ 2193.986773] LustreError: 26545:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558783 with bad export cookie 11797782520125416232 [ 2193.986782] LustreError: 11541:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558783 with bad export cookie 11797782520125416232 [ 2193.986820] LustreError: 26545:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558783 with bad export cookie 11797782520125416232 [ 2194.090540] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2205.154357] Lustre: server umount lustre-MDT0001 complete [ 2216.285822] Lustre: server umount lustre-OST0000 complete [ 2218.017833] Lustre: server umount lustre-OST0001 complete [ 2218.019528] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2218.023190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 2220.270045] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2220.275965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2237.363829] LustreError: 13a-8: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2253.367949] LustreError: 7253:0:(/home/green/git/lustre-release/lustre/mgc/mgc_request.c:253:do_config_log_add()) MGC192.168.202.142@tcp: failed processing log, type 4: rc = -110Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2286.036629] Lustre: Failing over lustre-OST0000Lustre: server umount lustre-OST0000 complete [ 2288.231470] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2288.236020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 2291.136305] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2291.142266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2308.241890] LustreError: 13a-8: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2324.244653] LustreError: 9215:0:(/home/green/git/lustre-release/lustre/mgc/mgc_request.c:253:do_config_log_add()) MGC192.168.202.142@tcp: failed processing log, type 4: rc = -110Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2356.479329] Lustre: DEBUG MARKER: == sanity-scrub test 20: Don't trigger OI scrub for irreparable oi repeatedly ========================================================== 23:42:24 (1717558944)Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local[ 2361.580549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2361.707450] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2362.452027] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all[ 2364.576909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2364.643449] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.[ 2364.647928] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck allLustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.[ 2366.655127] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2370.764728] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2372.516260] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2375.540826] Lustre: DEBUG MARKER: Using TIMEOUT=20Lustre: Failing over lustre-MDT0000 [ 2382.687850] Lustre: server umount lustre-MDT0000 completeLustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2384.692727] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2384.699100] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.[ 2384.770436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2384.794896] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2384.840652] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to completeLustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2385.583279] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2388.286965] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2389.869621] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2397.767828] Lustre: lustre-MDT0000: trigger OI scrub by RPC for [0x200003ab1:0x2:0x0]/16843009 with flags 0x4a: rc = 0 [ 2398.483821] Lustre: DEBUG MARKER: == sanity-scrub test 21: don't hang MDS recovery when failed to get update log ========================================================== 23:43:06 (1717558986) [ 2398.544013] Lustre: Failing over lustre-MDT0000 [ 2399.466887] Lustre: server umount lustre-MDT0000 complete [ 2399.467646] LustreError: 9221:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558988 with bad export cookie 11797782520125438751 [ 2399.467802] Lustre: Failing over lustre-MDT0001 [ 2399.467807] LustreError: 12616:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558988 with bad export cookie 11797782520125438751 [ 2399.467825] LustreError: 9222:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558988 with bad export cookie 11797782520125438751 [ 2399.467831] LustreError: 12616:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558988 with bad export cookie 11797782520125438751 [ 2399.467848] LustreError: 9222:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717558988 with bad export cookie 11797782520125438751 [ 2399.575270] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2400.274237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2401.946219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2409.467822] Lustre: server umount lustre-MDT0001 complete [ 2409.580399] LustreError: 18572:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8800a73e4a80 x1800988770381184/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 2410.355995] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2411.299754] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2412.527923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2412.590661] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2412.590724] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 2412.590819] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2412.590849] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2412.590850] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2413.333324] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2413.337499] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2414.344152] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2415.582703] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 200 [ 2415.590584] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558989/real 1717558989] req@ffff8800943b5880 x1800988770380416/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559005 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2415.598112] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558989/real 1717558989] req@ffff8800943b5f80 x1800988770380544/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559005 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2415.620570] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558989/real 1717558989] req@ffff8800943b5c00 x1800988770380480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559005 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2417.613338] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558989/real 1717558989] req@ffff8800943b4380 x1800988770380608/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559005 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2419.641581] LustreError: 18275:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/update_trans.c:1082:top_trans_stop()) lustre-MDT0000-osp-MDT0001: stop trans failed: rc = -20 [ 2419.645011] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558993/real 1717558993] req@ffff8800943b5180 x1800988770380800/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559009 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2419.645572] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558993/real 1717558993] req@ffff8800943b6300 x1800988770380864/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559009 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2419.652559] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558993/real 1717558993] req@ffff8800943b4a80 x1800988770380736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559009 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2422.613083] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558993/real 1717558993] req@ffff8800943b4700 x1800988770380672/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559009 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2422.613091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2422.613209] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2422.613247] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.612782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.631655] Lustre: 18573:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558998/real 1717558998] req@ffff8800943b6a00 x1800988770380992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559014 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2424.644628] Lustre: 18574:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558998/real 1717558998] req@ffff8800943b6d80 x1800988770381056/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559014 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2424.650546] Lustre: 18575:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558998/real 1717558998] req@ffff8800943b7480 x1800988770381120/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559014 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2427.066080] Lustre: 18576:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558998/real 1717558998] req@ffff8800943b6680 x1800988770380928/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559014 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2427.620888] Lustre: server umount lustre-MDT0000 complete [ 2427.620895] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2427.620932] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2427.620935] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.193103] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.193844] LustreError: 12616:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717559018 with bad export cookie 11797782520125440347 [ 2429.193847] LustreError: 9223:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717559018 with bad export cookie 11797782520125440347 [ 2429.193903] LustreError: 9222:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717559018 with bad export cookie 11797782520125440347 [ 2429.193940] LustreError: 9222:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717559018 with bad export cookie 11797782520125440347 [ 2429.193944] LustreError: 9223:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717559018 with bad export cookie 11797782520125440347 [ 2429.293180] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2441.467587] Lustre: server umount lustre-MDT0001 complete [ 2453.723193] Lustre: server umount lustre-OST0000 complete [ 2455.389268] device-mapper: core: cleaned up [ 2457.963181] Lustre: server umount lustre-OST0001 complete [ 2458.433188] Key type lgssc unregistered [ 2458.494876] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 2460.541109] device-mapper: uevent: version 1.0.3 [ 2460.542227] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 2464.664367] LNet: 21883:0:(/home/green/git/lustre-release/lnet/lnet/lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2468.440823] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 2468.593326] alg: No test for adler32 (adler32-zlib) [ 2469.345560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2470.996761] Key type lgssc registered [ 2476.098733] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2480.312148] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2482.655964] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 2482.657805] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 2482.661248] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2485.191050] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 2485.195348] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2488.416044] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local[ 2491.479816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2491.497054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'[ 2491.505622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.628378] Lustre: lustre-MDT0000: new disk, initializing [ 2493.418986] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all[ 2498.275185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2498.304655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2498.408719] Lustre: lustre-MDT0001: new disk, initializing [ 2499.378988] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck allLDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2504.926638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2504.949564] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2504.957703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2505.042610] Lustre: lustre-OST0000: new disk, initializing [ 2506.701490] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck allLDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2511.247654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2511.262538] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2511.265492] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2511.287895] Lustre: lustre-OST0001: new disk, initializing [ 2512.498355] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck allLustre: DEBUG MARKER: Using TIMEOUT=20 [ 2527.600455] Lustre: DEBUG MARKER: == sanity-scrub test complete, duration 2438 sec ========= 23:45:16 (1717559116)LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2528.474911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to completeLustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete[ 2530.309201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to completeLustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: server umount lustre-MDT0000 completeLustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.[ 2535.317012] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 27058:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717559126 with bad export cookie 6397620468011966064LustreError: 28211:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717559126 with bad export cookie 6397620468011966064 [ 2537.205777] LustreError: 29418:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717559126 with bad export cookie 6397620468011966064 [ 2537.205875] LustreError: 27057:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717559126 with bad export cookie 6397620468011966064 [ 2537.205924] LustreError: 27057:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:2593:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1717559126 with bad export cookie 6397620468011966064 [ 2537.335278] LustreError: 166-1: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2549.553314] Lustre: server umount lustre-MDT0001 complete [ 2561.743858] Lustre: server umount lustre-OST0000 complete [ 2563.397930] device-mapper: core: cleaned up [ 2565.940506] Lustre: server umount lustre-OST0001 complete [ 2566.362863] Key type lgssc unregistered [ 2566.423761] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local