[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 513008931 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.542074] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.544331] pid_max: default: 32768 minimum: 301 [ 0.546031] Security Framework initialized [ 0.547101] SELinux: Initializing. [ 0.548123] SELinux: Starting in permissive mode [ 0.549871] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.554119] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.556973] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.559203] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.561787] Initializing cgroup subsys memory [ 0.563104] Initializing cgroup subsys devices [ 0.564483] Initializing cgroup subsys freezer [ 0.565784] Initializing cgroup subsys net_cls [ 0.567014] Initializing cgroup subsys blkio [ 0.568187] Initializing cgroup subsys perf_event [ 0.569447] Initializing cgroup subsys hugetlb [ 0.570649] Initializing cgroup subsys pids [ 0.571788] Initializing cgroup subsys net_prio [ 0.574600] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.578030] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.580255] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.582751] tlb_flushall_shift: 6 [ 0.583798] FEATURE SPEC_CTRL Present [ 0.585103] FEATURE IBPB_SUPPORT Present [ 0.586236] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.588589] Spectre V2 : Vulnerable [ 0.590191] Speculative Store Bypass: Vulnerable [ 0.594016] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.605836] ACPI: Core revision 20130517 [ 0.610782] ACPI: All ACPI Tables successfully acquired [ 0.613093] ftrace: allocating 30294 entries in 119 pages [ 0.686222] Enabling x2apic [ 0.688068] Enabled x2apic [ 0.689347] Switched APIC routing to physical x2apic. [ 0.692767] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.694839] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.699323] TSC deadline timer enabled [ 0.701522] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.703915] ... version: 2 [ 0.704870] ... bit width: 48 [ 0.705969] ... generic registers: 4 [ 0.707489] ... value mask: 0000ffffffffffff [ 0.711204] ... max period: 00007fffffffffff [ 0.712677] ... fixed-purpose events: 3 [ 0.713763] ... event mask: 000000070000000f [ 0.715310] KVM setup paravirtual spinlock [ 0.727495] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.742663] KVM setup async PF for cpu 1 [ 0.745589] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.752169] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.722724] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.760088] KVM setup async PF for cpu 2 [ 0.760096] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.764091] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.771595] Brought up 4 CPUs [ 0.772492] smpboot: Max logical packages: 1 [ 0.773724] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.775323] KVM setup async PF for cpu 3 [ 0.775331] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.799171] devtmpfs: initialized [ 0.800345] x86/mm: Memory block size: 128MB [ 0.807276] EVM: security.selinux [ 0.808291] EVM: security.ima [ 0.809068] EVM: security.capability [ 0.814198] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.817383] NET: Registered protocol family 16 [ 0.819292] cpuidle: using governor haltpoll [ 0.821172] ACPI: bus type PCI registered [ 0.822504] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.824772] PCI: Using configuration type 1 for base access [ 0.826635] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.843919] ACPI: Added _OSI(Module Device) [ 0.845144] ACPI: Added _OSI(Processor Device) [ 0.846435] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.847582] ACPI: Added _OSI(Processor Aggregator Device) [ 0.849280] ACPI: Added _OSI(Linux-Dell-Video) [ 0.851255] ACPI: EC: Look up EC in DSDT [ 0.854396] ACPI: Interpreter enabled [ 0.855689] ACPI: (supports S0 S3 S4 S5) [ 0.856839] ACPI: Using IOAPIC for interrupt routing [ 0.858425] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.860939] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.869688] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.871770] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.873899] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.876192] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.880553] acpiphp: Slot [2] registered [ 0.881841] acpiphp: Slot [3] registered [ 0.883444] acpiphp: Slot [4] registered [ 0.886311] acpiphp: Slot [5] registered [ 0.888379] acpiphp: Slot [6] registered [ 0.889554] acpiphp: Slot [7] registered [ 0.890638] acpiphp: Slot [8] registered [ 0.891699] acpiphp: Slot [9] registered [ 0.892780] acpiphp: Slot [10] registered [ 0.894005] acpiphp: Slot [11] registered [ 0.895203] acpiphp: Slot [12] registered [ 0.896644] acpiphp: Slot [13] registered [ 0.897904] acpiphp: Slot [14] registered [ 0.899277] acpiphp: Slot [15] registered [ 0.900757] acpiphp: Slot [16] registered [ 0.902038] acpiphp: Slot [17] registered [ 0.903582] acpiphp: Slot [18] registered [ 0.905387] acpiphp: Slot [19] registered [ 0.907138] acpiphp: Slot [20] registered [ 0.908520] acpiphp: Slot [21] registered [ 0.909834] acpiphp: Slot [22] registered [ 0.910995] acpiphp: Slot [23] registered [ 0.912306] acpiphp: Slot [24] registered [ 0.913656] acpiphp: Slot [25] registered [ 0.915176] acpiphp: Slot [26] registered [ 0.916476] acpiphp: Slot [27] registered [ 0.917670] acpiphp: Slot [28] registered [ 0.918902] acpiphp: Slot [29] registered [ 0.921770] acpiphp: Slot [30] registered [ 0.923790] acpiphp: Slot [31] registered [ 0.926066] PCI host bridge to bus 0000:00 [ 0.929983] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.931889] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.933783] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.936364] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.938786] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.940847] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.942428] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.943260] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.944112] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.951458] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.954345] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.957645] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.960776] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.966098] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.972190] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.973044] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.977359] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.981661] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.984817] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.994699] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.996405] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.997683] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 1.005074] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 1.008895] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 1.031740] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 1.052022] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 1.063124] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 1.069338] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 1.093970] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 1.108465] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 1.120036] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 1.131739] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 1.159790] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 1.167365] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.176972] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.183907] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.201471] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 1.208450] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.217966] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.228582] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.252809] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 1.268826] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.281200] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.291906] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.315402] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 1.320703] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 1.327502] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 1.329847] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 1.337395] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 1.338907] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 1.344710] vgaarb: loaded [ 1.345777] SCSI subsystem initialized [ 1.346686] ACPI: bus type USB registered [ 1.348397] usbcore: registered new interface driver usbfs [ 1.354541] usbcore: registered new interface driver hub [ 1.361077] usbcore: registered new device driver usb [ 1.363626] PCI: Using ACPI for IRQ routing [ 1.365162] PCI: pci_cache_line_size set to 64 bytes [ 1.365461] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 1.365470] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 1.365472] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 1.365950] NetLabel: Initializing [ 1.366905] NetLabel: domain hash size = 128 [ 1.368215] NetLabel: protocols = UNLABELED CIPSOv4 [ 1.369846] NetLabel: unlabeled traffic allowed by default [ 1.372515] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.373902] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.380820] amd_nb: Cannot enumerate AMD northbridges [ 1.388164] Switched to clocksource kvm-clock [ 1.409215] pnp: PnP ACPI init [ 1.410179] ACPI: bus type PNP registered [ 1.411253] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 1.411325] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 1.411362] pnp 00:02: [dma 2] [ 1.411390] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 1.411458] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.411519] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.411583] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.411908] pnp: PnP ACPI: found 6 devices [ 1.413169] ACPI: bus type PNP unregistered [ 1.440992] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.441001] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.441005] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.441007] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.441009] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 1.441125] NET: Registered protocol family 2 [ 1.443110] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 1.448709] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.453182] TCP: Hash tables configured (established 32768 bind 32768) [ 1.458700] TCP: reno registered [ 1.459674] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.462562] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.467101] NET: Registered protocol family 1 [ 1.473076] RPC: Registered named UNIX socket transport module. [ 1.477700] RPC: Registered udp transport module. [ 1.481340] RPC: Registered tcp transport module. [ 1.482404] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.491739] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.497504] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.501507] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.508009] PCI: CLS 0 bytes, default 64 [ 1.513564] Unpacking initramfs... [ 4.032957] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 4.036486] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.038372] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 4.041985] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 4.044769] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 4.046609] RAPL PMU: hw unit of domain package 2^-0 Joules [ 4.047915] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 4.060627] cryptomgr_test (52) used greatest stack depth: 14480 bytes left [ 4.064545] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 4.066616] Initialise system trusted keyring [ 4.101940] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 4.104066] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 4.113324] zpool: loaded [ 4.114329] zbud: loaded [ 4.115538] VFS: Disk quotas dquot_6.6.0 [ 4.117570] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 4.123538] NFS: Registering the id_resolver key type [ 4.124843] Key type id_resolver registered [ 4.126804] Key type id_legacy registered [ 4.128163] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 4.131392] Key type big_key registered [ 4.133431] SELinux: Registering netfilter hooks [ 4.147530] cryptomgr_test (58) used greatest stack depth: 14048 bytes left [ 4.154193] cryptomgr_test (63) used greatest stack depth: 13984 bytes left [ 4.156512] NET: Registered protocol family 38 [ 4.158265] Key type asymmetric registered [ 4.159341] Asymmetric key parser 'x509' registered [ 4.160690] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 4.164528] io scheduler noop registered [ 4.165577] io scheduler deadline registered (default) [ 4.167111] io scheduler cfq registered [ 4.168384] io scheduler mq-deadline registered [ 4.169843] io scheduler kyber registered [ 4.173881] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 4.176323] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 4.178634] intel_idle: does not run on family 6 model 62 [ 4.178769] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.181242] ACPI: Power Button [PWRF] [ 4.184183] GHES: HEST is not enabled! [ 4.264154] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 4.346090] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 4.540489] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 4.615608] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 4.795500] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.832366] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 4.859533] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 4.865124] Non-volatile memory driver v1.3 [ 4.866365] Linux agpgart interface v0.103 [ 4.871808] crash memory driver: version 1.1 [ 4.873402] nbd: registered device at major 43 [ 4.884450] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 4.884505] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 4.884547] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 4.884586] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 4.884623] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 4.893896] virtio_blk virtio1: [vda] 67296 512-byte logical blocks (34.4 MB/32.8 MiB) [ 4.902530] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 4.902567] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 4.902603] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 4.902633] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 4.902663] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 4.916642] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.925323] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 4.925359] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 4.925389] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 4.925419] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 4.925453] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 4.937767] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.947252] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 4.947290] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 4.947320] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 4.947351] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 4.947385] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 4.957012] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.970996] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 4.971033] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 4.971104] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 4.971136] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 4.971166] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 4.979518] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.988514] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 4.988563] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 4.988599] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 4.988635] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 4.988670] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 5.009489] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.029796] rdac: device handler registered [ 5.037108] hp_sw: device handler registered [ 5.041311] emc: device handler registered [ 5.043010] libphy: Fixed MDIO Bus: probed [ 5.055223] tsc: Refined TSC clocksource calibration: 2399.978 MHz [ 5.055450] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 5.055476] ehci-pci: EHCI PCI platform driver [ 5.055523] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 5.055530] ohci-pci: OHCI PCI platform driver [ 5.055548] uhci_hcd: USB Universal Host Controller Interface driver [ 5.055724] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.064020] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.064075] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.064760] mousedev: PS/2 mouse device common for all mice [ 5.069453] rtc_cmos 00:05: RTC can wake from S4 [ 5.070638] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 5.070874] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.077416] hidraw: raw HID events driver (C) Jiri Kosina [ 5.077681] usbcore: registered new interface driver usbhid [ 5.077682] usbhid: USB HID core driver [ 5.080180] drop_monitor: Initializing network drop monitor service [ 5.080273] Netfilter messages via NETLINK v0.30. [ 5.080356] TCP: cubic registered [ 5.080364] Initializing XFRM netlink socket [ 5.080682] NET: Registered protocol family 10 [ 5.086107] NET: Registered protocol family 17 [ 5.086377] Key type dns_resolver registered [ 5.087659] mce: Using 10 MCE banks [ 5.089858] Loading compiled-in X.509 certificates [ 5.090891] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 5.090952] registered taskstats version 1 [ 5.097219] modprobe (72) used greatest stack depth: 13456 bytes left [ 5.102475] Key type trusted registered [ 5.110871] Key type encrypted registered [ 5.110939] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 5.113027] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 5.125534] rtc_cmos 00:05: setting system clock to 2024-04-10 14:56:38 UTC (1712760998) [ 5.129285] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.205525] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 5.216241] Write protecting the kernel read-only data: 12288k [ 5.218911] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 5.224760] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 5.239756] random: systemd: uninitialized urandom read (16 bytes read) [ 5.244132] random: systemd: uninitialized urandom read (16 bytes read) [ 5.247905] random: systemd: uninitialized urandom read (16 bytes read) [ 5.256326] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 5.261229] systemd[1]: Detected virtualization kvm. [ 5.262838] systemd[1]: Detected architecture x86-64. [ 5.264512] systemd[1]: Running in initial RAM disk. [ 5.270016] systemd[1]: No hostname configured. [ 5.271542] systemd[1]: Set hostname to . [ 5.273182] random: systemd: uninitialized urandom read (16 bytes read) [ 5.275777] systemd[1]: Initializing machine ID from random generator. [ 5.382317] dracut-rootfs-g (86) used greatest stack depth: 13264 bytes left [ 5.391926] random: systemd: uninitialized urandom read (16 bytes read) [ 5.397998] random: systemd: uninitialized urandom read (16 bytes read) [ 5.400569] random: systemd: uninitialized urandom read (16 bytes read) [ 5.404446] random: systemd: uninitialized urandom read (16 bytes read) [ 5.416653] random: systemd: uninitialized urandom read (16 bytes read) [ 5.419670] random: systemd: uninitialized urandom read (16 bytes read) [ 5.440715] systemd[1]: Created slice Root Slice. [ 5.445915] systemd[1]: Listening on Journal Socket. [ 5.450371] systemd[1]: Listening on udev Kernel Socket. [ 5.456617] systemd[1]: Reached target Timers. [ 5.459898] systemd[1]: Reached target Swap. [ 5.462882] systemd[1]: Listening on udev Control Socket. [ 5.471601] systemd[1]: Reached target Sockets. [ 5.482282] systemd[1]: Reached target Local File Systems. [ 5.486912] systemd[1]: Created slice System Slice. [ 5.505743] systemd[1]: Starting Setup Virtual Console... [ 5.529053] systemd[1]: Starting Journal Service... [ 5.543191] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 5.579331] systemd[1]: Starting Load Kernel Modules... [ 5.596493] systemd[1]: Reached target Slices. [ 5.628399] systemd[1]: Starting dracut cmdline hook... [ 5.659603] systemd[1]: Started Setup Virtual Console. [ 5.718289] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 5.738650] systemd[1]: Started Load Kernel Modules. [ 5.756469] systemd[1]: Started Journal Service. [ 6.184283] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 6.810220] random: fast init done [ 7.044602] libata version 3.00 loaded. [ 7.068685] ata_piix 0000:00:01.1: version 2.13 [ 7.092219] scsi host0: ata_piix [ 7.108936] scsi host1: ata_piix [ 7.112857] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.117745] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 7.448164] ip (322) used greatest stack depth: 13080 bytes left [ 7.547746] ip (345) used greatest stack depth: 11584 bytes left [ 11.451800] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 12.391850] systemd-journald[102]: Received SIGTERM from PID 1 (systemd). [ 12.994392] SELinux: Disabled at runtime. [ 12.995682] SELinux: Unregistering netfilter hooks [ 13.187569] ip_tables: (C) 2000-2006 Netfilter Core Team [ 13.192838] systemd[1]: Inserted module 'ip_tables' [ 14.825463] systemd-journald[569]: Received request to flush runtime journal from PID 1 [ 15.806261] hrtimer: interrupt took 3693042 ns [ 16.054663] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 16.244206] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 16.321363] cryptd: max_cpu_qlen set to 1000 [ 16.327177] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 16.691287] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 16.875322] AVX version of gcm_enc/dec engaged. [ 16.880690] AES CTR mode by8 optimization enabled [ 16.931949] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 16.936584] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 17.692582] EDAC MC: Ver: 3.0.0 [ 17.757155] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 17.757169] EDAC sbridge: Ver: 1.1.2 [ 23.512548] mount.nfs (781) used greatest stack depth: 10704 bytes left [ 37.590339] device-mapper: uevent: version 1.0.3 [ 37.594497] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 44.845262] libcfs: loading out-of-tree module taints kernel. [ 44.847321] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 44.881537] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 51.194446] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 51.473385] alg: No test for adler32 (adler32-zlib) [ 52.225152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.404267] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 52.631521] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 52.633931] LNet: Accept secure, port 988 [ 54.191216] Key type lgssc registered [ 54.631008] Lustre: Echo OBD driver; http://www.lustre.org/ [ 58.918177] icp: module license 'CDDL' taints kernel. [ 58.919735] Disabling lock debugging due to kernel taint [ 61.981202] ZFS: Loaded module v0.8.6-1, ZFS pool version 5000, ZFS filesystem version 5 [ 65.887610] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 71.718507] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 74.819151] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 74.830553] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 78.382249] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 78.392511] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 83.748455] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 88.360188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 88.391423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 88.402376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 89.520287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 89.536374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 89.575287] Lustre: lustre-MDT0000: new disk, initializing [ 89.628399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 89.638723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 89.686767] mount.lustre (6544) used greatest stack depth: 10256 bytes left [ 90.665885] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 95.709089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 95.747490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 95.780977] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 95.792346] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 95.794773] Lustre: Skipped 1 previous similar message [ 95.832495] Lustre: lustre-MDT0001: new disk, initializing [ 95.853580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 95.871205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 95.875314] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 96.898831] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 102.877413] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 102.882849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 102.908324] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 102.912237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 102.994780] Lustre: lustre-OST0000: new disk, initializing [ 102.998631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 103.022778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 103.567024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 103.572708] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 103.607557] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 104.660294] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 109.696652] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 109.702463] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 109.728189] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 109.734803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 109.771019] Lustre: lustre-OST0001: new disk, initializing [ 109.772550] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 109.793662] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 111.706520] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 117.510362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 117.513666] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 117.528311] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 117.616001] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 123.397563] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 129.164921] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing check_logdir /tmp/testlogs/ [ 130.285451] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing yml_node [ 131.625918] Lustre: DEBUG MARKER: Client: 2.15.62.1 [ 132.403791] Lustre: DEBUG MARKER: MDS: 2.15.62.1 [ 133.255265] Lustre: DEBUG MARKER: OSS: 2.15.62.1 [ 133.491922] random: crng init done [ 133.774466] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Apr 10 10:58:47 EDT 2024 [ 135.568262] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball 110 [ 135.990229] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 147.550665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 147.550852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 147.559598] Lustre: Skipped 3 previous similar messages [ 152.556479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 152.559203] Lustre: Skipped 6 previous similar messages [ 152.978418] Lustre: server umount lustre-MDT0000 complete [ 154.239818] LustreError: 6554:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761147 with bad export cookie 13687421438527908166 [ 154.240893] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 154.248210] LustreError: 6554:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 157.564285] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 157.569358] Lustre: Skipped 1 previous similar message [ 157.569438] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 157.569741] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 159.568265] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 159.572922] LustreError: Skipped 2 previous similar messages [ 160.361125] Lustre: server umount lustre-MDT0001 complete [ 167.760283] Lustre: server umount lustre-OST0000 complete [ 175.310711] Lustre: server umount lustre-OST0001 complete [ 177.506169] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 180.171358] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 183.510447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 186.024403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 188.444086] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 188.450184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 191.113881] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 191.129802] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 194.156282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 194.192832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 194.330660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 194.346425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 194.388389] Lustre: lustre-MDT0000: new disk, initializing [ 194.428773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.448721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 196.044502] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 201.409642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 201.443908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 201.485078] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 201.647504] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 201.678015] mount.lustre (16885) used greatest stack depth: 9632 bytes left [ 202.951306] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 206.186391] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 207.272980] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 210.047545] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 210.055072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 210.079421] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 210.085211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 210.196226] Lustre: lustre-OST0000: new disk, initializing [ 210.197862] Lustre: Skipped 1 previous similar message [ 210.200098] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 210.204304] Lustre: Skipped 2 previous similar messages [ 210.242187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 210.247179] Lustre: Skipped 1 previous similar message [ 211.996876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 212.002139] Lustre: Skipped 1 previous similar message [ 212.004223] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 212.035554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 212.132893] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 214.983857] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 216.239485] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 216.336513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 217.540963] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 217.640034] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 220.504870] Lustre: server umount lustre-OST0000 complete [ 222.043814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 222.045318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 222.045319] Lustre: Skipped 3 previous similar messages [ 222.055034] Lustre: Skipped 1 previous similar message [ 227.052985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 227.058132] Lustre: Skipped 2 previous similar messages [ 231.766580] Lustre: server umount lustre-MDT0000 complete [ 232.059932] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 232.065680] LustreError: Skipped 1 previous similar message [ 233.109894] LustreError: 15945:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761226 with bad export cookie 13687421438527909608 [ 233.112512] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 233.119622] LustreError: 15945:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 243.277809] Lustre: server umount lustre-MDT0001 complete [ 246.049580] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 11:00:39 (1712761239) [ 246.959141] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 249.724864] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 11:00:43 (1712761243) [ 252.218769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 252.418173] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 252.459760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 253.604390] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 256.594785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 257.748525] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 259.245212] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 260.071595] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 262.387605] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 262.392508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 264.299392] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 265.647862] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 287.564347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 287.564385] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 287.564686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 287.574279] Lustre: Skipped 1 previous similar message [ 289.054750] Lustre: server umount lustre-OST0000 complete [ 291.756503] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 291.760023] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.198995] Lustre: server umount lustre-MDT0000 complete [ 297.339368] LustreError: 21319:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761290 with bad export cookie 13687421438527910462 [ 297.341527] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 297.348445] LustreError: 21319:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 297.471573] Lustre: server umount lustre-MDT0001 complete [ 300.699547] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 301.294571] Key type lgssc unregistered [ 301.369806] LNet: 24967:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 301.373225] LNet: Removed LNI 192.168.202.152@tcp [ 310.275755] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 310.454315] alg: No test for adler32 (adler32-zlib) [ 311.206165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 311.303029] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 311.365747] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 311.368341] LNet: Accept secure, port 988 [ 312.919252] Key type lgssc registered [ 313.210085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 315.739696] loop: module loaded [ 315.886106] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 318.110304] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 318.113528] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 321.557844] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 321.577274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 321.583648] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 322.679725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 322.690498] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 322.728339] Lustre: lustre-MDT0000: new disk, initializing [ 322.750589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 322.760126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 323.835688] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 329.297123] LDISKFS-fs (dm-5): file extents enabled, maximum tree depth=5 [ 329.303817] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 329.323525] LDISKFS-fs (dm-5): file extents enabled, maximum tree depth=5 [ 329.327732] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 329.414806] Lustre: lustre-OST07c6: new disk, initializing [ 329.416711] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 329.419313] Lustre: Skipped 1 previous similar message [ 329.443342] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 329.668037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 329.672375] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 329.684701] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 331.057738] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 394.763875] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 394.768688] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 399.771626] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 400.317192] Lustre: server umount lustre-OST07c6 complete [ 402.999456] Lustre: server umount lustre-MDT0000 complete [ 406.586264] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 407.080920] Key type lgssc unregistered [ 407.143515] LNet: 32281:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 407.146027] LNet: Removed LNI 192.168.202.152@tcp [ 414.140555] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 414.313836] alg: No test for adler32 (adler32-zlib) [ 415.069146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 415.130073] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 415.178591] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 415.180312] LNet: Accept secure, port 988 [ 416.708651] Key type lgssc registered [ 416.963042] Lustre: Echo OBD driver; http://www.lustre.org/ [ 419.041400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 419.048813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 420.201933] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 420.215612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 421.010685] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 423.429935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 423.555820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 424.381012] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 425.556528] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 426.223992] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 428.079191] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 428.084206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 428.197196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 429.610539] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 430.200542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 430.858999] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 440.219726] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 440.222925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.227746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 443.212597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 443.212735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 444.995033] Lustre: server umount lustre-OST0000 complete [ 448.219843] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 448.220225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 448.220226] Lustre: Skipped 1 previous similar message [ 448.229499] Lustre: Skipped 1 previous similar message [ 452.104842] Lustre: server umount lustre-MDT0000 complete [ 453.166149] LustreError: 4125:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761446 with bad export cookie 7794829219223957281 [ 453.167482] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 453.174213] LustreError: 4125:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 453.278112] Lustre: server umount lustre-MDT0001 complete [ 456.508773] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 457.130169] Key type lgssc unregistered [ 457.208411] LNet: 5192:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 457.211637] LNet: Removed LNI 192.168.202.152@tcp [ 466.192900] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 466.334269] alg: No test for adler32 (adler32-zlib) [ 467.086135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 467.162288] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 467.234553] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 467.238058] LNet: Accept secure, port 988 [ 468.774338] Key type lgssc registered [ 469.043113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 471.402676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 471.411166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 472.593749] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 472.613416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 473.615691] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 476.279749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 476.417672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 477.354670] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 478.683935] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 479.471553] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 481.634785] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 481.641966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 481.778811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 482.775547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 483.302458] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 484.539197] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 491.419740] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 491.423083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 491.427575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 495.708287] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 495.788512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.793831] Lustre: Skipped 1 previous similar message [ 497.302903] Lustre: server umount lustre-MDT0000 complete [ 498.364091] LustreError: 7076:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761491 with bad export cookie 13977031144569283184 [ 498.365298] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 498.373085] LustreError: 7076:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 500.709357] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.717524] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.52@tcp (stopping) [ 500.720189] Lustre: Skipped 3 previous similar messages [ 500.795846] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 502.799567] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.799651] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 502.799652] Lustre: Skipped 1 previous similar message [ 502.808890] LustreError: Skipped 1 previous similar message [ 504.469121] Lustre: server umount lustre-MDT0001 complete [ 515.076648] LustreError: 7962:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.52@tcp arrived at 1712761508 with bad export cookie 13977031144569284024 [ 515.082809] LustreError: 7962:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 521.700119] Lustre: server umount lustre-OST0000 complete [ 526.220879] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 526.725419] Key type lgssc unregistered [ 526.787414] LNet: 11133:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 526.789879] LNet: Removed LNI 192.168.202.152@tcp [ 534.791976] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 534.917978] alg: No test for adler32 (adler32-zlib) [ 535.669135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 535.729350] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 535.776092] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 535.777689] LNet: Accept secure, port 988 [ 537.305236] Key type lgssc registered [ 537.532833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 539.662327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 539.670799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 540.835523] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.847977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 541.676758] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 544.021788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 544.126169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 544.954361] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 546.097487] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 546.744690] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 548.592695] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 548.597848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 548.694221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 550.058520] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 550.695238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 551.209085] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 558.700256] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 558.700454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 558.701065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 559.983249] Lustre: server umount lustre-OST0000 complete [ 573.724331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 573.724857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 573.724858] Lustre: Skipped 1 previous similar message [ 573.732674] Lustre: Skipped 2 previous similar messages [ 577.970480] Lustre: server umount lustre-MDT0000 complete [ 578.731579] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.737398] LustreError: Skipped 1 previous similar message [ 579.037779] LustreError: 16060:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761572 with bad export cookie 11266802894963790541 [ 579.038365] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 579.046588] LustreError: 16060:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 585.164258] Lustre: server umount lustre-MDT0001 complete [ 588.170814] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 588.685843] Key type lgssc unregistered [ 588.750521] LNet: 16859:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 588.754128] LNet: Removed LNI 192.168.202.152@tcp [ 597.245225] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 597.373242] alg: No test for adler32 (adler32-zlib) [ 598.124196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 598.190027] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 598.243090] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 598.245082] LNet: Accept secure, port 988 [ 599.777178] Key type lgssc registered [ 600.003802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 602.095365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 602.101811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 603.246448] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 603.262577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 604.082498] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 606.513226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 606.648010] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 607.522439] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 608.695346] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 609.269694] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 611.004905] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 611.008634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 611.098454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 612.334749] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 613.098749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 613.447239] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 619.970565] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 620.860575] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 626.101426] LNetError: 120-3: Refusing connection from 192.168.202.152 for 127.0.0.2@tcp: No matching NI [ 626.104395] LNetError: 17984:0:(socklnd_cb.c:1795:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 626.108063] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 631.277900] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2024-04-1011h07m04s [ 631.679199] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 634.604468] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 636.667699] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 636.670243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 636.674429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 637.132956] LNetError: 120-3: Refusing connection from 192.168.202.152 for 127.0.0.2@tcp: No matching NI [ 637.135911] LNetError: 17981:0:(socklnd_cb.c:1795:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 637.141811] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 638.269436] Lustre: server umount lustre-MDT0000 complete [ 639.169884] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 639.620932] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 641.003901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 641.046739] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 641.120189] LNetError: 120-3: Refusing connection from 192.168.202.152 for 127.0.0.2@tcp: No matching NI [ 641.123219] LNetError: 17982:0:(socklnd_cb.c:1795:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 641.126307] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 641.131897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 641.150426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:193) [ 641.962337] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 642.867687] Lustre: DEBUG MARKER: Wait for df (31506) ... [ 645.022772] Lustre: DEBUG MARKER: done [ 646.123803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 646.124349] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 646.124780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 646.124781] Lustre: Skipped 3 previous similar messages [ 646.125452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 646.137682] Lustre: Skipped 3 previous similar messages [ 647.760956] Lustre: server umount lustre-OST0000 complete [ 650.792027] Lustre: server umount lustre-MDT0000 complete [ 651.131656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 651.136020] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 651.136711] LNetError: 120-3: Refusing connection from 192.168.202.152 for 127.0.0.2@tcp: No matching NI [ 651.136741] LNetError: 17983:0:(socklnd_cb.c:1795:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 651.136758] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 651.151407] LustreError: Skipped 1 previous similar message [ 651.772016] LustreError: 18738:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761645 with bad export cookie 4802679098077337834 [ 651.772937] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 651.780111] LustreError: 18738:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 657.872352] Lustre: server umount lustre-MDT0001 complete [ 660.743657] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 661.218988] Key type lgssc unregistered [ 661.279510] LNet: 24097:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 661.282010] LNet: Removed LNI 192.168.202.152@tcp [ 669.738568] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 674.154255] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 674.326239] alg: No test for adler32 (adler32-zlib) [ 675.077212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 675.141034] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 675.188853] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 675.190959] LNet: Accept secure, port 988 [ 676.723210] Key type lgssc registered [ 676.947134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 679.113686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 680.799707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 682.325942] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 682.329394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 683.968454] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 683.971943] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 685.781345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 685.795774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 685.800426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 686.870770] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 686.880286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 686.913358] Lustre: lustre-MDT0000: new disk, initializing [ 686.930745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 686.936131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 687.704823] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 691.141179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 691.157197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 691.182062] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 691.193786] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 691.195890] Lustre: Skipped 1 previous similar message [ 691.226796] Lustre: lustre-MDT0001: new disk, initializing [ 691.249061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 691.259267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 691.263023] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 692.014428] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 694.287625] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 694.877274] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 696.605518] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 696.608999] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 696.625256] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 696.628877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 696.699134] Lustre: lustre-OST0000: new disk, initializing [ 696.701678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 696.719622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 697.938455] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 700.187713] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 700.965129] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 704.829887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 704.833343] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 704.852131] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 705.070917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 705.870340] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 705.922309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 709.852062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 709.852243] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 709.859748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 709.861913] Lustre: Skipped 1 previous similar message [ 712.493013] Lustre: server umount lustre-OST0000 complete [ 714.859775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 714.860188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 714.866307] Lustre: Skipped 2 previous similar messages [ 719.614164] Lustre: server umount lustre-MDT0000 complete [ 719.867555] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 719.871773] LustreError: Skipped 1 previous similar message [ 720.650756] LustreError: 32497:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761714 with bad export cookie 15493724155099446031 [ 720.652300] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 720.658718] LustreError: 32497:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 720.757189] Lustre: server umount lustre-MDT0001 complete [ 724.625964] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 11:08:38 (1712761718) [ 726.531794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 726.662858] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 726.679953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 727.478324] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 729.710284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 730.664892] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 731.812222] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 732.470192] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 734.299378] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 734.303245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 734.397369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 734.399678] Lustre: Skipped 1 previous similar message [ 735.657645] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 736.776102] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 744.811934] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2024-04-1011h08m58s [ 745.200545] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 746.953400] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 747.593154] Lustre: DEBUG MARKER: Creating a test file and stat it [ 747.604491] LustreError: 1606:0:(mdt_handler.c:2320:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 751.716089] LNetError: 120-3: Refusing connection from 192.168.202.152 for 127.0.0.2@tcp: No matching NI [ 751.719488] LNetError: 26680:0:(socklnd_cb.c:1795:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 751.722639] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 767.619424] Lustre: lustre-MDT0000: Client 7eec8d59-553b-4cfa-b47c-8301870a9661 (at 192.168.202.52@tcp) reconnecting [ 787.608156] LustreError: 1606:0:(mdt_handler.c:2320:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 787.985197] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 788.618796] Lustre: DEBUG MARKER: done [ 790.476318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 790.476507] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 790.476509] LustreError: Skipped 1 previous similar message [ 790.476833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 790.476835] Lustre: Skipped 1 previous similar message [ 790.490291] Lustre: Skipped 1 previous similar message [ 795.483950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 795.486568] Lustre: Skipped 2 previous similar messages [ 795.818870] Lustre: server umount lustre-OST0000 complete [ 799.916201] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 799.919273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 799.924121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 800.492900] LNetError: 120-3: Refusing connection from 192.168.202.152 for 127.0.0.2@tcp: No matching NI [ 800.496361] LNetError: 26677:0:(socklnd_cb.c:1795:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 800.499746] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 803.025569] Lustre: server umount lustre-MDT0000 complete [ 804.110010] LustreError: 2478:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761797 with bad export cookie 15493724155099446871 [ 804.111424] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 804.118664] LustreError: 2478:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 804.228232] Lustre: server umount lustre-MDT0001 complete [ 807.354976] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 807.854288] Key type lgssc unregistered [ 807.922457] LNet: 6253:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.925440] LNet: Removed LNI 192.168.202.152@tcp [ 816.297418] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 820.631822] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 820.767775] alg: No test for adler32 (adler32-zlib) [ 821.519154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 821.575360] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 821.622739] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 821.624832] LNet: Accept secure, port 988 [ 823.151182] Key type lgssc registered [ 823.347911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 825.297397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 826.817176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 828.221711] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 828.224747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 829.610455] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 829.614251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 831.286509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 831.301346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 831.306409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 832.369292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 832.377273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 832.409274] Lustre: lustre-MDT0000: new disk, initializing [ 832.423769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 832.428782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 833.138152] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 836.585434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 836.601447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 836.627334] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 836.639625] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 836.641773] Lustre: Skipped 1 previous similar message [ 836.672783] Lustre: lustre-MDT0001: new disk, initializing [ 836.694096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 836.702969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 836.706510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 837.503373] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 839.943546] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 840.551312] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 842.351213] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 842.356012] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 842.381209] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 842.385320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 842.461933] Lustre: lustre-OST0000: new disk, initializing [ 842.464255] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 842.484301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 843.807811] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 846.135459] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 846.954722] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 849.724887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 849.728215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 849.743943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 850.052344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 850.799763] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 850.847675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 854.747802] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 854.750743] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 854.755472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 854.758371] Lustre: Skipped 1 previous similar message [ 857.389273] Lustre: server umount lustre-OST0000 complete [ 859.755736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 859.760160] Lustre: Skipped 1 previous similar message [ 859.762001] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 859.763632] Lustre: Skipped 1 previous similar message [ 864.406362] Lustre: server umount lustre-MDT0000 complete [ 864.763556] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 864.769092] LustreError: Skipped 1 previous similar message [ 865.382920] LustreError: 14650:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761858 with bad export cookie 9278450713821168286 [ 865.384549] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 865.392211] LustreError: 14650:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 865.481152] Lustre: server umount lustre-MDT0001 complete [ 868.861746] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 11:11:02 (1712761862) [ 869.195180] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 871.036572] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 11:11:04 (1712761864) [ 873.044988] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 873.792556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 873.899962] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 873.914657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 874.168198] Lustre: Failing over lustre-MDT0000 [ 874.170259] LustreError: 16945:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 874.174187] LustreError: 16945:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 874.176953] LustreError: 16945:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -5 [ 874.267620] Lustre: server umount lustre-MDT0000 complete [ 877.034858] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 11:11:10 (1712761870) [ 878.622269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 878.734860] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 878.745462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 879.445601] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 881.532781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 882.337095] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 883.319300] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 883.874008] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 885.479557] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 885.482785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 886.734573] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 887.752118] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 895.428157] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 901.579839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 901.580393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 901.585998] Lustre: Skipped 3 previous similar messages [ 902.780185] Lustre: server umount lustre-MDT0000 complete [ 903.715121] LustreError: 18129:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761897 with bad export cookie 9278450713821169399 [ 903.716722] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 903.722292] LustreError: 18129:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 903.822026] Lustre: server umount lustre-MDT0001 complete [ 904.610484] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 905.744078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 907.868504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 914.587502] LustreError: 8848:0:(client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8800a161fb80 x1795960932957376/t0(0) o250->MGC192.168.202.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 914.690085] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 914.707098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 914.709374] Lustre: Skipped 2 previous similar messages [ 914.727809] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 915.483611] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 917.674182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 917.751337] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 917.786122] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:41 to 0x280000400:65) [ 918.555014] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 919.625033] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 920.198220] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 920.779780] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 921.707184] Lustre: 8850:0:(client.c:2340:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1712761899/real 1712761899] req@ffff88012d179f80 x1795960932957184/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1712761915 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 923.777173] Lustre: 8851:0:(client.c:2340:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1712761901/real 1712761901] req@ffff8800a161e300 x1795960932957248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1712761917 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 927.455177] Lustre: 8852:0:(client.c:2340:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1712761904/real 1712761904] req@ffff8800a161e680 x1795960932957312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1712761920 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 927.771637] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 927.774650] LustreError: Skipped 1 previous similar message [ 927.776067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 927.780619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 927.783113] Lustre: Skipped 2 previous similar messages [ 929.780160] Lustre: 8850:0:(client.c:2340:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1712761907/real 1712761907] req@ffff8800a161f100 x1795960932957440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1712761923 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 932.508340] Lustre: server umount lustre-MDT0000 complete [ 933.517006] LustreError: 20000:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761926 with bad export cookie 9278450713821171604 [ 933.518115] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 933.525759] LustreError: 20000:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 933.629815] Lustre: server umount lustre-MDT0001 complete [ 935.361890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 937.888784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 942.803481] LustreError: 8848:0:(client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88012ba42680 x1795960932969088/t0(0) o250->MGC192.168.202.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 942.898213] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 942.913785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 942.916131] Lustre: Skipped 1 previous similar message [ 942.927490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000bd0:3 to 0x280000bd0:33) [ 943.661204] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 945.799881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 945.871321] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 945.875666] Lustre: Skipped 2 previous similar messages [ 945.906066] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:41 to 0x280000400:97) [ 946.645522] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 947.660720] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 948.202220] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 950.897259] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 950.899327] Lustre: Skipped 1 previous similar message [ 951.895147] Lustre: 8849:0:(client.c:2340:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1712761929/real 1712761929] req@ffff880129f6ea00 x1795960932968896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1712761945 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 955.899713] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 955.900200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 955.900201] Lustre: Skipped 3 previous similar messages [ 960.529465] Lustre: server umount lustre-MDT0000 complete [ 960.907722] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 960.911410] LustreError: Skipped 2 previous similar messages [ 961.494900] LustreError: 18129:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712761954 with bad export cookie 9278450713821173291 [ 961.495843] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 961.502313] LustreError: 18129:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 961.593973] Lustre: server umount lustre-MDT0001 complete [ 963.225395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 964.043392] Lustre: DEBUG MARKER: files compared the same [ 974.597578] Lustre: server umount lustre-OST0000 complete [ 978.930458] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 979.397759] Key type lgssc unregistered [ 979.458431] LNet: 28799:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.461383] LNet: Removed LNI 192.168.202.152@tcp [ 986.936836] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 987.093220] alg: No test for adler32 (adler32-zlib) [ 987.845199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 987.900910] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 987.945312] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 987.947678] LNet: Accept secure, port 988 [ 989.473216] Key type lgssc registered [ 989.676566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 991.487077] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 991.492034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.615640] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 992.627639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.362001] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug +malloc all [ 995.430240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 995.508848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 996.237966] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug +malloc all [ 997.235234] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 997.767420] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 999.416822] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 999.420936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 999.490889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1000.642056] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug +malloc all [ 1001.624442] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1003.490821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000bd0:35 to 0x280000bd0:65) [ 1005.490675] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:41 to 0x280000400:129) [ 1010.507787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1010.507841] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1010.507973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1010.516382] Lustre: Skipped 1 previous similar message [ 1011.636865] Lustre: server umount lustre-OST0000 complete [ 1015.515513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1015.515682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1015.515684] Lustre: Skipped 1 previous similar message [ 1015.523239] Lustre: Skipped 1 previous similar message [ 1018.645530] Lustre: server umount lustre-MDT0000 complete [ 1019.640192] LustreError: 30719:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762013 with bad export cookie 14910144876703762902 [ 1019.640468] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1019.647496] LustreError: 30719:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1019.735713] Lustre: server umount lustre-MDT0001 complete [ 1022.696796] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1023.186834] Key type lgssc unregistered [ 1023.249458] LNet: 1886:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.252372] LNet: Removed LNI 192.168.202.152@tcp [ 1031.264984] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1031.408814] alg: No test for adler32 (adler32-zlib) [ 1032.160195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1032.220441] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 1032.268908] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1032.270817] LNet: Accept secure, port 988 [ 1033.798250] Key type lgssc registered [ 1034.009510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1035.924277] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1035.928289] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1035.932467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1052.992362] LustreError: 3757:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.202.152@tcp: failed processing log, type 1: rc = -5 [ 1088.997295] LustreError: 3757:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.202.152@tcp: failed processing log, type 4: rc = -110 [ 1239.013210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1240.228501] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1241.289254] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1243.268097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1243.293947] Lustre: *** cfs_fail_loc=706, val=0*** [ 1250.923496] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1250.940005] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1250.954391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000bd0:35 to 0x280000bd0:97) [ 1251.737565] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1252.938003] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1252.942720] LustreError: Skipped 1 previous similar message [ 1254.058524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1254.157887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1254.172964] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:41 to 0x280000400:161) [ 1254.924661] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1256.029282] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1256.657104] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1259.147766] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1259.150856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1259.155425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1263.302027] Lustre: server umount lustre-OST0000 complete [ 1269.163697] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1269.167498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1269.171934] Lustre: Skipped 1 previous similar message [ 1269.173631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1269.175801] Lustre: Skipped 1 previous similar message [ 1270.363408] Lustre: server umount lustre-MDT0000 complete [ 1271.367088] LustreError: 6859:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762264 with bad export cookie 3139359943110563952 [ 1271.368280] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1271.377062] LustreError: 6859:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1271.497783] Lustre: server umount lustre-MDT0001 complete [ 1273.472781] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1273.929644] Key type lgssc unregistered [ 1273.996393] LNet: 7655:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1273.999086] LNet: Removed LNI 192.168.202.152@tcp [ 1281.978609] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1282.087735] alg: No test for adler32 (adler32-zlib) [ 1282.839130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1282.899782] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 1282.947882] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1282.949678] LNet: Accept secure, port 988 [ 1284.477262] Key type lgssc registered [ 1284.712083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1286.683273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1286.688996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1288.517888] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1290.562877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1290.692318] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1290.714549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1291.508601] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1293.780888] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1293.784745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1293.875013] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1293.879360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1293.884465] LustreError: Skipped 2 previous similar messages [ 1295.121057] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1297.344854] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1297.348889] Lustre: MGS: Client 21b813c3-99a9-4a4b-843a-741215f52db9 (at 0@lo) reconnecting [ 1297.416198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1297.430484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000bd0:35 to 0x280000bd0:129) [ 1298.227856] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1301.415261] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:41 to 0x280000400:193) [ 1306.427968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1306.428041] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1306.428577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1306.439744] Lustre: Skipped 1 previous similar message [ 1307.569624] Lustre: server umount lustre-OST0000 complete [ 1310.618507] Lustre: server umount lustre-MDT0000 complete [ 1311.435654] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1311.435802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1311.436257] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1311.701730] Lustre: server umount lustre-MDT0001 complete [ 1312.726517] Lustre: server umount MGS complete [ 1317.085193] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:18:30 (1712762310) [ 1324.080107] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 1325.540857] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1327.651885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1329.261674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1330.765281] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1330.768457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1332.294864] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1332.298433] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1334.030442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1334.051166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1334.799712] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1338.052795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1338.072894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1338.182359] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1338.196372] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1338.234570] Lustre: lustre-MDT0001: new disk, initializing [ 1338.255010] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1338.263335] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1338.266096] Lustre: 19574:0:(lod_sub_object.c:966:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1338.270070] LustreError: 19574:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1339.049562] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1342.546222] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1342.550153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1342.567875] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1342.572117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1342.658323] Lustre: lustre-OST0000: new disk, initializing [ 1342.660462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1342.677282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1343.924486] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1344.398670] Lustre: 20700:0:(client.c:1513:after_reply()) @@@ resending request on EINPROGRESS req@ffff8800873b7800 x1795961416334528/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1712762353 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 [ 1346.201477] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1347.922670] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1347.928124] Lustre: MGS: Client 5c55e7b8-16f9-4488-80b3-452b834e3756 (at 0@lo) reconnecting [ 1347.940292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1347.945988] LustreError: 18707:0:(mgs_handler.c:264:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1347.955438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1347.995508] Lustre: lustre-MDT0000: new disk, initializing [ 1348.034468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1348.832684] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1349.042864] Lustre: 21624:0:(client.c:1513:after_reply()) @@@ resending request on EINPROGRESS req@ffff8800873b5500 x1795961416337152/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1712762358 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 [ 1349.061581] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1353.406665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1353.410701] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1354.051814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1354.066011] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1372.731945] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1372.732293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1372.732295] Lustre: Skipped 1 previous similar message [ 1372.741925] Lustre: Skipped 2 previous similar messages [ 1377.468546] Lustre: server umount lustre-OST0000 complete [ 1382.747683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.747804] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1382.748143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1382.748145] Lustre: Skipped 1 previous similar message [ 1382.757213] Lustre: Skipped 1 previous similar message [ 1384.488995] Lustre: server umount lustre-MDT0000 complete [ 1391.606532] Lustre: server umount lustre-MDT0001 complete [ 1392.595385] Lustre: server umount MGS complete [ 1396.764649] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:19:50 (1712762390) [ 1401.360837] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1401.833472] Key type lgssc unregistered [ 1401.902409] LNet: 25012:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1401.905136] LNet: Removed LNI 192.168.202.152@tcp [ 1406.498801] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1406.640443] alg: No test for adler32 (adler32-zlib) [ 1407.391175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1407.449335] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 1407.496455] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1407.498233] LNet: Accept secure, port 988 [ 1409.027223] Key type lgssc registered [ 1409.232089] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1411.088412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1411.094259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1412.222525] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1412.233615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1413.025947] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1415.228932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1415.327920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1415.347998] mount.lustre (27114) used greatest stack depth: 9584 bytes left [ 1416.080569] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1417.096228] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1417.675312] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1419.365059] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1419.368305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1419.460340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1420.640647] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1421.642173] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1422.461995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1430.475986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1430.476295] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1430.476646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1430.482680] Lustre: Skipped 1 previous similar message [ 1431.690913] Lustre: server umount lustre-OST0000 complete [ 1435.356238] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1435.363656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1435.375455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1435.379460] Lustre: Skipped 1 previous similar message [ 1438.736847] Lustre: server umount lustre-MDT0000 complete [ 1439.800793] LustreError: 29087:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762433 with bad export cookie 8448856860993149025 [ 1439.802475] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1439.809566] LustreError: 29087:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1439.919318] Lustre: server umount lustre-MDT0001 complete [ 1442.860004] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1443.400069] Key type lgssc unregistered [ 1443.469448] LNet: 29883:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1443.471610] LNet: Removed LNI 192.168.202.152@tcp [ 1447.829030] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1447.936984] alg: No test for adler32 (adler32-zlib) [ 1448.688175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1448.746367] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 1448.791691] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1448.793351] LNet: Accept secure, port 988 [ 1450.321216] Key type lgssc registered [ 1450.524504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1451.968248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1451.968623] LustreError: 30952:0:(tgt_mount.c:2178:server_fill_super()) cfs_race id 716 sleeping [ 1451.975743] Lustre: Skipped 1 previous similar message [ 1451.977944] LustreError: 30951:0:(tgt_mount.c:2178:server_fill_super()) cfs_fail_race id 716 waking [ 1451.981722] LustreError: 30952:0:(tgt_mount.c:2178:server_fill_super()) cfs_fail_race id 716 awake: rc=4987 [ 1451.984755] LustreError: 30952:0:(tgt_mount.c:2190:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1451.987376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1451.996128] LustreError: 30952:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1453.112692] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1453.123389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1455.084370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1455.165329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1455.904592] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1456.888451] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing lsmod [ 1457.615830] LustreError: 32340:0:(tgt_mount.c:2178:server_fill_super()) cfs_race id 716 sleeping [ 1457.615846] LustreError: 32339:0:(tgt_mount.c:2178:server_fill_super()) cfs_fail_race id 716 waking [ 1457.618722] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1457.620685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1457.630388] LustreError: 32340:0:(tgt_mount.c:2178:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 1457.633827] LustreError: 32340:0:(tgt_mount.c:2190:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1457.637772] LustreError: 32340:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1457.716748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1458.461144] LustreError: 31023:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1458.463937] LustreError: 31023:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1458.466113] LustreError: 31023:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 0, failed: rc = -5 [ 1460.171717] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1460.174035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1460.177934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1462.731893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1462.732067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1462.737676] Lustre: Skipped 2 previous similar messages [ 1464.527174] Lustre: server umount lustre-MDT0000 complete [ 1464.733247] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1464.737010] LustreError: Skipped 2 previous similar messages [ 1465.535919] LustreError: 32579:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762458 with bad export cookie 14744198427801457706 [ 1465.537504] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1465.542800] LustreError: 32579:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1466.752481] LustreError: 31553:0:(client.c:1281:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88012b412680 x1795961590389824/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 1466.866953] Lustre: server umount lustre-MDT0001 complete [ 1477.946722] Lustre: server umount lustre-OST0000 complete [ 1480.151429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1480.291410] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1480.308075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1481.054396] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1483.271084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1484.131467] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1485.202755] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1485.830983] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1487.625035] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1487.629081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1487.728398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1487.731783] Lustre: Skipped 1 previous similar message [ 1488.975111] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1489.729070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1490.051885] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1497.739847] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1497.740133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1497.740135] Lustre: Skipped 1 previous similar message [ 1497.740546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1497.740548] Lustre: Skipped 1 previous similar message [ 1498.937373] Lustre: server umount lustre-OST0000 complete [ 1501.945691] Lustre: server umount lustre-MDT0000 complete [ 1502.747633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1502.748166] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1502.758262] Lustre: Skipped 2 previous similar messages [ 1502.947006] LustreError: 2188:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762496 with bad export cookie 14744198427801458357 [ 1502.948742] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1502.960881] LustreError: 2188:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1503.059506] Lustre: server umount lustre-MDT0001 complete [ 1505.969768] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1506.480846] Key type lgssc unregistered [ 1506.549457] LNet: 4945:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1506.551712] LNet: Removed LNI 192.168.202.152@tcp [ 1514.401090] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1514.551287] alg: No test for adler32 (adler32-zlib) [ 1515.302187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1515.364121] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 1515.413640] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1515.415384] LNet: Accept secure, port 988 [ 1516.943261] Key type lgssc registered [ 1517.147278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1519.058615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1519.063728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1520.194616] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1520.205954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1520.984446] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1523.258814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1523.361440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1524.125246] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1525.212006] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1525.794336] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1527.542828] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1527.546286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1527.646419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1528.883819] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1529.646837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1529.935700] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1536.380088] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 1537.659637] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1537.662308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1537.669025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1537.672440] Lustre: Skipped 1 previous similar message [ 1539.563420] Lustre: server umount lustre-OST0000 complete [ 1542.643872] Lustre: server umount lustre-MDT0000 complete [ 1542.667721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1542.668103] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1542.674923] Lustre: Skipped 2 previous similar messages [ 1543.633836] LustreError: 6833:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762537 with bad export cookie 11925230065793545398 [ 1543.635267] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1543.641676] LustreError: 6833:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1543.743297] Lustre: server umount lustre-MDT0001 complete [ 1546.667598] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1547.193363] Key type lgssc unregistered [ 1547.259479] LNet: 10607:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1547.262117] LNet: Removed LNI 192.168.202.152@tcp [ 1552.186168] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1552.353307] alg: No test for adler32 (adler32-zlib) [ 1553.104144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1553.169603] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 1553.220629] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1553.222417] LNet: Accept secure, port 988 [ 1554.753221] Key type lgssc registered [ 1554.952740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1556.825476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1556.830059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1557.953953] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1557.965675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1558.745839] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1560.891683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1560.986213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1561.713527] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1562.724624] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1563.324246] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1565.034066] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1565.037450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1565.135293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1566.395779] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1567.485294] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1568.137259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1576.139927] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1576.139983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1576.140429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1577.692494] Lustre: server umount lustre-OST0000 complete [ 1580.730402] Lustre: server umount lustre-MDT0000 complete [ 1581.003688] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1581.005926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1581.009095] Lustre: Skipped 1 previous similar message [ 1581.010790] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1581.707812] LustreError: 11838:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762575 with bad export cookie 13846024487206225033 [ 1581.709211] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1581.717109] LustreError: 11838:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1587.836666] Lustre: server umount lustre-MDT0001 complete [ 1590.660952] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1591.114075] Key type lgssc unregistered [ 1591.181460] LNet: 15474:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1591.185624] LNet: Removed LNI 192.168.202.152@tcp [ 1598.863822] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1599.011381] alg: No test for adler32 (adler32-zlib) [ 1599.762227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1599.827743] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 1599.874759] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1599.877692] LNet: Accept secure, port 988 [ 1601.407274] Key type lgssc registered [ 1601.615266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1603.547348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1603.552588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.681003] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1604.692251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1605.440220] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1607.596241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1607.704536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1608.449126] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1609.466064] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1610.008903] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1611.679006] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1611.683499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1611.785778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1613.043324] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1614.111877] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1614.787405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1622.216469] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 1623.189447] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 1623.191782] Lustre: Skipped 2 previous similar messages [ 1624.256913] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 1624.258678] Lustre: Skipped 2 previous similar messages [ 1625.194447] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1625.200210] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1632.695355] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 1632.697780] Lustre: Skipped 2 previous similar messages [ 1642.556382] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.52@tcp 192.168.0.[2,10]@tcp [ 1644.827799] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1644.830807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1644.835455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1647.836366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1647.836609] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1647.836625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1647.856588] Lustre: Skipped 1 previous similar message [ 1649.620696] Lustre: server umount lustre-OST0000 complete [ 1652.764249] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1652.768547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1652.776069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1656.685678] Lustre: server umount lustre-MDT0000 complete [ 1657.672930] LustreError: 17354:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762651 with bad export cookie 12582087731062789829 [ 1657.674360] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1657.679876] LustreError: 17354:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1657.792490] Lustre: server umount lustre-MDT0001 complete [ 1660.669393] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1661.126187] Key type lgssc unregistered [ 1661.189466] LNet: 22649:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1661.192963] LNet: Removed LNI 192.168.202.152@tcp [ 1671.540540] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1671.666399] alg: No test for adler32 (adler32-zlib) [ 1672.420129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1672.478160] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 1672.523345] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1672.524708] LNet: Accept secure, port 988 [ 1674.053189] Key type lgssc registered [ 1674.253956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1676.112029] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1676.117076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1677.244741] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1677.254867] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1677.255715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1677.262705] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.202.52@tcp 192.168.0.[2,10]@tcp [ 1678.015604] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1680.179743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1680.257884] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1680.262301] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.52@tcp 192.168.0.[2,10]@tcp [ 1680.271596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1681.006803] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1682.111142] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1682.659006] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1684.349560] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1684.352874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1684.450556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1685.627060] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1686.623061] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1687.452467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:167 to 0x240000401:193) [ 1695.467899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1695.468115] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1695.468312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1695.474239] Lustre: Skipped 1 previous similar message [ 1697.100338] Lustre: server umount lustre-OST0000 complete [ 1700.136400] Lustre: server umount lustre-MDT0000 complete [ 1700.299779] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1700.302808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1700.307557] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1701.129957] LustreError: 25109:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762694 with bad export cookie 6445151449612903218 [ 1701.131439] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1701.138656] LustreError: 25109:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1707.239906] Lustre: server umount lustre-MDT0001 complete [ 1710.138986] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1710.615089] Key type lgssc unregistered [ 1710.679449] LNet: 28817:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.682166] LNet: Removed LNI 192.168.202.152@tcp [ 1718.223986] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 1722.368891] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1722.483153] alg: No test for adler32 (adler32-zlib) [ 1723.234168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1723.294619] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 1723.339325] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1723.340797] LNet: Accept secure, port 988 [ 1724.866176] Key type lgssc registered [ 1725.063451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1727.141120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1728.771502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1730.296063] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1730.300498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1731.836283] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1731.839657] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1733.623778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1733.636957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1733.640612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1734.717212] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1734.726274] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1734.763392] Lustre: lustre-MDT0000: new disk, initializing [ 1734.782442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1734.790806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1735.574725] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1739.045728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1739.061232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1739.089931] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1739.101987] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1739.103867] Lustre: Skipped 1 previous similar message [ 1739.136860] Lustre: lustre-MDT0001: new disk, initializing [ 1739.155987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1739.164129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1739.170604] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1739.916804] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1742.246909] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1742.864670] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1744.557869] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1744.561893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1744.581358] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1744.585154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1744.668945] Lustre: lustre-OST0000: new disk, initializing [ 1744.670426] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1744.689004] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1744.710715] mount.lustre (2809) used greatest stack depth: 9552 bytes left [ 1745.914696] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1747.552483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1747.555238] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1747.571410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1748.190495] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1748.992734] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1749.041293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1749.765824] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1749.807440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1752.572000] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1752.572099] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1752.572699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1752.577707] Lustre: Skipped 1 previous similar message [ 1756.333473] Lustre: server umount lustre-OST0000 complete [ 1757.579760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1757.580071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1757.580072] Lustre: Skipped 1 previous similar message [ 1757.584496] Lustre: Skipped 1 previous similar message [ 1762.587721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1762.589540] Lustre: Skipped 2 previous similar messages [ 1763.342750] Lustre: server umount lustre-MDT0000 complete [ 1764.319892] LustreError: 781:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762757 with bad export cookie 12786244240588159743 [ 1764.321104] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1764.325392] LustreError: 781:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1764.438360] Lustre: server umount lustre-MDT0001 complete [ 1766.558410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1766.701762] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1766.716394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1767.488778] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1769.726567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1770.605354] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1771.626382] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1772.169661] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1773.881559] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1773.885056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1773.971843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1773.973779] Lustre: Skipped 1 previous similar message [ 1775.188993] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1776.214124] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1776.995494] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1777.047550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1777.806743] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1780.893578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 1783.632318] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1783.635114] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1783.653731] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1783.657067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1783.690310] Lustre: lustre-OST0001: new disk, initializing [ 1783.691895] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1783.707013] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1784.969977] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1787.461491] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1787.668365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1787.671638] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1787.687633] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1788.530775] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1789.320606] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1789.371257] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1789.961334] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800aa990800.ost_server_uuid 50 [ 1790.340770] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8800aa990800.ost_server_uuid in FULL state after 0 sec [ 1792.684135] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1792.684653] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1792.685002] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1792.692272] LustreError: Skipped 2 previous similar messages [ 1793.654727] Lustre: server umount lustre-OST0001 complete [ 1794.987734] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1794.989949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1794.993259] Lustre: Skipped 1 previous similar message [ 1796.727654] Lustre: server umount lustre-OST0000 complete [ 1799.851842] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1799.855211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1799.860775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1799.863033] Lustre: Skipped 2 previous similar messages [ 1807.823780] Lustre: server umount lustre-MDT0000 complete [ 1808.837836] LustreError: 10634:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762802 with bad export cookie 12786244240588160583 [ 1808.838948] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1808.847667] LustreError: 10634:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1808.959649] Lustre: server umount lustre-MDT0001 complete [ 1814.147504] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1814.638591] Key type lgssc unregistered [ 1814.703355] LNet: 12036:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.705392] LNet: Removed LNI 192.168.202.152@tcp [ 1823.216625] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 1827.527357] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1827.670782] alg: No test for adler32 (adler32-zlib) [ 1828.425155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1828.487943] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 1828.539780] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1828.541748] LNet: Accept secure, port 988 [ 1830.072196] Key type lgssc registered [ 1830.293221] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1832.503816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1834.218902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1835.709981] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1835.713079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1837.336772] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1837.340438] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1839.137986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1839.155742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1839.160277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1840.241966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1840.249843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1840.288806] Lustre: lustre-MDT0000: new disk, initializing [ 1840.307736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1840.313889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1841.079411] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1844.585846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1844.605330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1844.628926] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1844.641795] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1844.643432] Lustre: Skipped 1 previous similar message [ 1844.678537] Lustre: lustre-MDT0001: new disk, initializing [ 1844.702566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1844.712044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1844.716512] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1845.475495] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1847.798497] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1848.362246] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1850.139705] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1850.143060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1850.163066] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1850.166733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1850.253429] Lustre: lustre-OST0000: new disk, initializing [ 1850.256441] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1850.275832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1851.544990] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1853.858787] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1854.695121] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1854.727041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1854.729959] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1854.745922] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1855.765490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1856.593537] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1856.646500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1859.755902] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1859.755961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1859.756551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1859.763798] LustreError: Skipped 1 previous similar message [ 1863.233116] Lustre: server umount lustre-OST0000 complete [ 1864.731708] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1864.733780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1864.738096] Lustre: Skipped 1 previous similar message [ 1864.740087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1864.741752] Lustre: Skipped 1 previous similar message [ 1869.771701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1869.773804] Lustre: Skipped 2 previous similar messages [ 1870.320226] Lustre: server umount lustre-MDT0000 complete [ 1871.315018] LustreError: 16816:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762864 with bad export cookie 16367705135618586807 [ 1871.316760] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1871.321545] LustreError: 16816:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1871.435277] Lustre: server umount lustre-MDT0001 complete [ 1874.967779] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 11:27:48 (1712762868) [ 1878.807791] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 1880.462123] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1882.774723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1884.430024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1885.968251] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1885.971018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1887.550975] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1887.555062] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1889.337756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1889.355433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1889.439621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1889.447080] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1889.484315] Lustre: lustre-MDT0000: new disk, initializing [ 1889.505825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1889.512401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1890.276420] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1893.751739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1893.766914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1893.792145] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1893.804695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1893.806417] Lustre: Skipped 1 previous similar message [ 1893.840006] Lustre: lustre-MDT0001: new disk, initializing [ 1893.864059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1893.873718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1893.877247] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1894.657300] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1896.980011] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1897.590063] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1899.379455] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1899.382619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1899.399575] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1899.402363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1900.744825] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1903.077639] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1903.177843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1903.181797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1903.196884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1908.862723] Lustre: Failing over lustre-OST0000 [ 1908.931361] Lustre: server umount lustre-OST0000 complete [ 1920.939381] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1920.942328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1921.033579] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1921.037887] Lustre: Skipped 1 previous similar message [ 1921.038169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1921.038170] Lustre: Skipped 1 previous similar message [ 1921.041597] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1921.045564] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1922.239617] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1923.209307] Lustre: Failing over lustre-MDT0000 [ 1923.212949] LustreError: 14627:0:(import.c:695:ptlrpc_connect_import_locked()) can't connect to a closed import [ 1923.228172] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 1923.241581] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 1923.241622] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 1923.269845] Lustre: server umount lustre-MDT0000 complete [ 1923.899718] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1923.903713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1923.910048] Lustre: Skipped 1 previous similar message [ 1923.912723] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1926.059746] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1926.063076] LustreError: Skipped 2 previous similar messages [ 1928.229526] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1929.059268] Lustre: 14630:0:(client.c:2340:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1712762906/real 1712762906] req@ffff8800a5eb5f80 x1795961987819904/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1712762922 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 1931.067672] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1931.071115] LustreError: Skipped 2 previous similar messages [ 1934.060142] Lustre: 14631:0:(client.c:2340:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1712762911/real 1712762911] req@ffff8800a5eb4a80 x1795961987820480/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1712762927 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 1935.350275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.385070] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1935.388896] Lustre: 24780:0:(mgc_request.c:1844:mgc_process_log()) MGC192.168.202.152@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 1935.445412] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 1935.470583] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1936.236630] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1938.245656] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1940.461388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 1940.468648] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 1940.483926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 1943.275762] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1943.278782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1943.283937] Lustre: Skipped 2 previous similar messages [ 1943.285988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1947.236375] Lustre: server umount lustre-OST0000 complete [ 1954.286629] Lustre: server umount lustre-MDT0000 complete [ 1955.276887] LustreError: 30113:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712762948 with bad export cookie 16367705135618588802 [ 1955.278369] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1955.285167] LustreError: 30113:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1955.402533] Lustre: server umount lustre-MDT0001 complete [ 1958.354139] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1958.857734] Key type lgssc unregistered [ 1958.925515] LNet: 30908:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1958.928248] LNet: Removed LNI 192.168.202.152@tcp [ 1966.902846] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1967.052117] alg: No test for adler32 (adler32-zlib) [ 1967.803180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1967.869029] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 1967.921658] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1967.922994] LNet: Accept secure, port 988 [ 1969.453254] Key type lgssc registered [ 1969.668520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1971.641886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1971.646893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1972.785432] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1972.796520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1973.595311] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1975.865972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1975.968483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1976.768538] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1977.849981] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1978.469212] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1980.287938] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1980.292060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1980.388742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1981.684832] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1982.795246] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1983.388869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:65) [ 2065.760612] Lustre: Failing over lustre-MDT0000 [ 2065.819930] Lustre: server umount lustre-MDT0000 complete [ 2066.091512] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2066.095501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2066.101597] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2069.052613] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2069.056954] LustreError: Skipped 2 previous similar messages [ 2071.531660] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2071.537293] LustreError: Skipped 2 previous similar messages [ 2076.539629] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2076.545398] LustreError: Skipped 3 previous similar messages [ 2077.984433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2078.011741] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2078.080450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2078.091638] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2078.836410] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2079.061227] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2083.084773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2083.091233] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2083.105578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:68 to 0x280000401:97) [ 2083.650818] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2084.033424] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2098.107725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.108611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2098.113274] Lustre: Skipped 4 previous similar messages [ 2101.997910] Lustre: server umount lustre-MDT0000 complete [ 2102.993833] LustreError: 320:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763096 with bad export cookie 12697102287456161229 [ 2102.995248] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2102.998711] LustreError: 320:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2103.115664] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2103.115758] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2103.116009] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2103.116010] Lustre: Skipped 2 previous similar messages [ 2108.123817] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2109.095694] Lustre: server umount lustre-MDT0001 complete [ 2116.159273] Lustre: server umount lustre-OST0000 complete [ 2118.247794] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 2119.927826] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2122.158694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2123.832179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2125.369522] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2125.374097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2126.957553] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2126.961900] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2128.740136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2128.759075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2128.832333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2128.840563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2128.877009] Lustre: lustre-MDT0000: new disk, initializing [ 2128.897398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2128.904496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2129.679393] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2133.244756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2133.268536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2133.292883] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2133.306184] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2133.307801] Lustre: Skipped 1 previous similar message [ 2133.343879] Lustre: lustre-MDT0001: new disk, initializing [ 2133.373597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2133.377341] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2134.166543] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2136.542793] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2137.158321] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2138.923098] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2138.926311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2138.944865] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2138.947607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2139.022497] Lustre: lustre-OST0000: new disk, initializing [ 2139.024705] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2139.041723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.043872] Lustre: Skipped 1 previous similar message [ 2140.289481] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2142.239855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2142.242131] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2142.258420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2142.608710] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2143.437158] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2143.488798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2144.305420] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2144.359797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2147.259843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2147.259937] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2147.260408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2147.265955] Lustre: Skipped 1 previous similar message [ 2150.938570] Lustre: server umount lustre-OST0000 complete [ 2152.267746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2152.268013] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2152.268015] Lustre: Skipped 1 previous similar message [ 2152.273745] Lustre: Skipped 1 previous similar message [ 2158.043585] Lustre: server umount lustre-MDT0000 complete [ 2159.067840] LustreError: 7665:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763152 with bad export cookie 12697102287456161950 [ 2159.069373] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2159.072880] LustreError: 7665:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2159.185239] Lustre: server umount lustre-MDT0001 complete [ 2162.820616] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:32:36 (1712763156) [ 2166.791874] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 2168.376088] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2170.617469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2172.317860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2173.811259] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2173.814712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2175.358407] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2175.361250] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2177.124740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2177.143243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2177.223556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2177.232199] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2177.267171] Lustre: lustre-MDT0000: new disk, initializing [ 2177.288164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2177.295439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2178.090766] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2181.609029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2181.630076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2181.653967] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2181.665428] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2181.667543] Lustre: Skipped 1 previous similar message [ 2181.704628] Lustre: lustre-MDT0001: new disk, initializing [ 2181.734523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2181.737464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2182.516221] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2184.877490] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2185.447769] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2187.182502] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2187.186420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2187.202341] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2187.205778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2188.607181] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2190.958649] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2194.910167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2194.912580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2194.931233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2199.932186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.936621] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2199.936986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2199.936988] Lustre: Skipped 3 previous similar messages [ 2199.944488] LustreError: Skipped 2 previous similar messages [ 2203.420008] Lustre: server umount lustre-OST0000 complete [ 2210.437689] Lustre: server umount lustre-MDT0000 complete [ 2211.450924] LustreError: 17649:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763204 with bad export cookie 12697102287456162790 [ 2211.451946] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2211.455922] LustreError: 17649:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2211.557247] Lustre: server umount lustre-MDT0001 complete [ 2215.098485] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:33:28 (1712763208) [ 2218.750063] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 2220.306515] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2222.468315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2224.143358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2225.693439] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2225.696432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2227.272033] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2227.275224] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2229.031069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2229.048276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2229.117655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2229.124609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2229.126719] Lustre: Skipped 1 previous similar message [ 2229.161953] Lustre: lustre-MDT0000: new disk, initializing [ 2229.163024] Lustre: Skipped 1 previous similar message [ 2229.179288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2229.180980] Lustre: Skipped 2 previous similar messages [ 2229.184881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2229.967409] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2233.489321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2233.509090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2233.615861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2234.379172] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2236.643602] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2237.245102] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2239.013395] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2239.017083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2239.034303] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2239.038441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2240.407824] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2242.688197] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2244.288898] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2244.310664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2249.307865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2249.308140] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2249.308518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2249.308519] Lustre: Skipped 5 previous similar messages [ 2249.323674] Lustre: Skipped 4 previous similar messages [ 2251.144733] Lustre: server umount lustre-OST0000 complete [ 2253.627824] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2253.631246] LustreError: Skipped 1 previous similar message [ 2254.193744] Lustre: server umount lustre-MDT0000 complete [ 2254.315610] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2254.320209] LustreError: Skipped 2 previous similar messages [ 2255.205064] LustreError: 23188:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763248 with bad export cookie 12697102287456163826 [ 2255.206090] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2255.214228] LustreError: 23188:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2255.317844] Lustre: server umount lustre-MDT0001 complete [ 2258.267996] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 2258.762296] Key type lgssc unregistered [ 2258.830511] LNet: 27194:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2258.833540] LNet: Removed LNI 192.168.202.152@tcp [ 2266.821284] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2266.977027] alg: No test for adler32 (adler32-zlib) [ 2267.728223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2267.788355] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 2267.839780] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 2267.842520] LNet: Accept secure, port 988 [ 2269.370220] Key type lgssc registered [ 2269.602425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2271.622263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2271.628435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2272.765661] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2272.776066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2273.581594] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2275.847592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2275.952252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2276.701876] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2277.771599] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2278.397858] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2280.247385] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2280.250821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2280.364577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2281.697439] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2282.790454] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2283.364771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2301.387747] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2301.388169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2301.388483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2305.992780] Lustre: server umount lustre-OST0000 complete [ 2310.987619] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2310.989612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2310.992324] Lustre: Skipped 1 previous similar message [ 2310.994078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2310.995440] Lustre: Skipped 1 previous similar message [ 2313.068149] Lustre: server umount lustre-MDT0000 complete [ 2314.155900] LustreError: 29072:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763307 with bad export cookie 2131127333588029875 [ 2314.157925] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2314.161195] LustreError: 29072:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2314.267906] Lustre: server umount lustre-MDT0001 complete [ 2317.261643] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 2317.815460] Key type lgssc unregistered [ 2317.886686] LNet: 32720:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2317.890690] LNet: Removed LNI 192.168.202.152@tcp [ 2325.961968] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2326.106580] alg: No test for adler32 (adler32-zlib) [ 2326.861150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2326.918956] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 2326.962898] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 2326.964212] LNet: Accept secure, port 988 [ 2328.491238] Key type lgssc registered [ 2328.719386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2330.757044] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2330.762302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2331.911732] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2331.923403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2332.751130] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2335.169837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2335.286800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2336.084695] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2337.216731] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2337.838822] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2339.726413] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2339.730467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2339.844338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2341.160129] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2341.843941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2342.259928] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2349.852103] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2349.852889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2349.853271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2351.048347] Lustre: server umount lustre-OST0000 complete [ 2352.173718] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff88012e38f000.ost_server_uuid 50 [ 2369.840893] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2379.883729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2379.887812] Lustre: Skipped 1 previous similar message [ 2379.888378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2379.888379] Lustre: Skipped 1 previous similar message [ 2384.891578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2384.894761] Lustre: Skipped 2 previous similar messages [ 2385.585155] Lustre: server umount lustre-MDT0000 complete [ 2386.569066] LustreError: 2173:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763379 with bad export cookie 17494770774756741666 [ 2386.570426] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2386.575737] LustreError: 2173:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2386.671483] Lustre: server umount lustre-MDT0001 complete [ 2390.156266] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:36:23 (1712763383) [ 2391.818633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2391.960754] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2391.976167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2392.770836] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2394.955742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2395.772430] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2396.697197] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2397.288730] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2399.180826] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2399.185625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2399.301213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2399.303564] Lustre: Skipped 1 previous similar message [ 2400.486804] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2401.564309] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2402.303317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2403.369904] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2403.373717] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2403.390418] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2403.394200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2403.430392] Lustre: lustre-OST0001: new disk, initializing [ 2403.432477] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2404.663130] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2405.379931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2405.383354] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2405.397517] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2406.919506] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2411.404121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2411.404495] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2411.404812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2411.412099] Lustre: Skipped 1 previous similar message [ 2413.588578] Lustre: server umount lustre-OST0000 complete [ 2414.832414] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2414.882998] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2415.686140] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2415.747350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2416.412104] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2416.416025] LustreError: Skipped 1 previous similar message [ 2421.420278] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2421.425808] LustreError: Skipped 2 previous similar messages [ 2425.419418] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2425.419456] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2425.419730] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2425.419731] Lustre: Skipped 2 previous similar messages [ 2425.427520] LustreError: Skipped 1 previous similar message [ 2426.427965] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.430939] LustreError: Skipped 2 previous similar messages [ 2431.462558] Lustre: server umount lustre-OST0001 complete [ 2435.099717] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2435.101851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2435.104752] Lustre: Skipped 1 previous similar message [ 2435.106663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2435.108410] Lustre: Skipped 3 previous similar messages [ 2438.608293] Lustre: server umount lustre-MDT0000 complete [ 2439.595812] LustreError: 6696:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763432 with bad export cookie 17494770774756742723 [ 2439.597088] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2439.602010] LustreError: 6696:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2439.718656] Lustre: server umount lustre-MDT0001 complete [ 2441.261759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2442.432282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2443.381956] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2443.385912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2444.540504] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2444.546247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2447.649814] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:37:21 (1712763441) [ 2449.274421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2449.357821] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2449.366988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2449.428227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2449.430940] Lustre: Skipped 1 previous similar message [ 2450.168888] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2452.252106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2452.276448] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2452.286673] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2453.082442] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2454.099082] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2454.662429] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2456.415558] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2456.419965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2456.500549] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2457.762043] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2457.826594] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 2458.784197] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2460.533274] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2460.537853] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2460.562384] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2461.862757] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2462.867948] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2465.628190] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2465.632165] Lustre: Skipped 1 previous similar message [ 2465.632951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2465.632952] Lustre: Skipped 2 previous similar messages [ 2469.490458] Lustre: server umount lustre-OST0000 complete [ 2472.539842] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2472.544144] LustreError: Skipped 1 previous similar message [ 2482.555928] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2482.556347] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2485.740389] Lustre: server umount lustre-OST0001 complete [ 2492.786141] Lustre: server umount lustre-MDT0000 complete [ 2493.762855] LustreError: 14061:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763487 with bad export cookie 17494770774756744067 [ 2493.764004] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2493.769144] LustreError: 14061:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2493.888169] Lustre: server umount lustre-MDT0001 complete [ 2495.324565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2496.272357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2497.246108] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2497.250192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2498.250579] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2498.254859] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2501.368681] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:38:14 (1712763494) [ 2505.073652] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 2506.498190] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2508.676116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2510.290398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2511.854001] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2511.860326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2513.455826] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2513.460224] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2515.210586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2515.227760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.307713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2515.316854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2515.349416] Lustre: lustre-MDT0000: new disk, initializing [ 2515.366298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2515.367968] Lustre: Skipped 3 previous similar messages [ 2515.371998] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2516.086453] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2519.466354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2519.486464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2519.509535] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2519.523398] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2519.525463] Lustre: Skipped 1 previous similar message [ 2519.556881] Lustre: lustre-MDT0001: new disk, initializing [ 2519.584266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2519.587586] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2520.363188] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2522.624285] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2523.180593] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2524.905261] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2524.908420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2524.929331] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2524.933013] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2525.010399] Lustre: lustre-OST0000: new disk, initializing [ 2525.012124] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2526.219444] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2528.442260] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2529.199424] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2530.360472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2530.363097] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2530.379269] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2531.274568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2531.998517] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2532.052603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2535.371986] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2535.372383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2535.372385] Lustre: Skipped 3 previous similar messages [ 2535.372688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2535.372689] Lustre: Skipped 10 previous similar messages [ 2535.388534] LustreError: Skipped 2 previous similar messages [ 2538.606902] Lustre: server umount lustre-OST0000 complete [ 2545.867262] Lustre: server umount lustre-MDT0000 complete [ 2546.812849] LustreError: 23704:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763540 with bad export cookie 17494770774756745474 [ 2546.814127] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2546.821472] LustreError: 23704:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2546.916786] Lustre: server umount lustre-MDT0001 complete [ 2549.023634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2549.151914] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.155828] LustreError: Skipped 6 previous similar messages [ 2549.170689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2549.173988] Lustre: Skipped 2 previous similar messages [ 2549.890675] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2552.108987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2552.944507] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2554.005028] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2554.577882] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2556.362052] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2556.364949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2557.767140] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2558.811813] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2559.617089] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2559.668814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2560.422344] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2563.506590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 2564.491912] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2570.102315] Lustre: server umount lustre-OST0000 complete [ 2571.433877] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2571.489754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2572.269073] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2572.321216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2595.743348] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2595.751524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2597.277842] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2598.461631] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2598.915982] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2605.773935] Lustre: server umount lustre-OST0000 complete [ 2606.882966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2606.884567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2606.884570] Lustre: Skipped 8 previous similar messages [ 2606.892007] Lustre: Skipped 6 previous similar messages [ 2612.930622] Lustre: server umount lustre-MDT0000 complete [ 2614.173739] LustreError: 28175:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763607 with bad export cookie 17494770774756746314 [ 2614.175695] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2614.185748] LustreError: 28175:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2614.354325] Lustre: server umount lustre-MDT0001 complete [ 2618.348283] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:40:11 (1712763611) [ 2620.014420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2620.140221] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2620.154677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2620.156150] Lustre: Skipped 3 previous similar messages [ 2620.963369] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2623.374397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2624.231895] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2625.277648] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2625.869024] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2627.715688] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2627.719295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2629.181823] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2630.345471] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2631.135261] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2631.187877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2631.924794] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2633.019464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 2634.927129] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2634.930006] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2634.944357] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2634.947133] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2634.981669] Lustre: lustre-OST0001: new disk, initializing [ 2634.983424] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2636.488122] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2636.797666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2636.800799] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2636.817217] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2639.170800] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2640.198259] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2640.257732] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2641.240635] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2641.300984] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2646.827575] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2646.829796] LustreError: Skipped 2 previous similar messages [ 2647.912011] Lustre: server umount lustre-OST0001 complete [ 2649.355953] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2649.399715] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2650.133994] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2650.180218] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2673.525627] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2673.530088] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2674.737258] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2676.023392] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2676.593793] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2683.144346] Lustre: server umount lustre-OST0001 complete [ 2688.604836] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2688.619153] LustreError: Skipped 18 previous similar messages [ 2700.447922] Lustre: server umount lustre-OST0000 complete [ 2707.467444] Lustre: server umount lustre-MDT0000 complete [ 2708.417790] LustreError: 2569:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763701 with bad export cookie 17494770774756747217 [ 2708.418980] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2708.425386] LustreError: 2569:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2714.545338] Lustre: server umount lustre-MDT0001 complete [ 2716.257275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2717.278512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2718.255901] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2718.260070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2719.230151] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2719.234438] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2722.236277] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:41:55 (1712763715) [ 2723.899438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2723.998976] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2724.007738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2724.800410] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2727.175674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2727.213242] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2728.039477] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2729.149296] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2729.749674] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2731.745282] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2731.748700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2731.814852] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2733.112067] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2734.126502] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2742.374916] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2742.379772] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2742.408780] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2743.820667] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2745.258948] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2746.871660] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 2747.927198] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2747.972123] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2748.754714] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2748.813832] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2749.637904] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800a80c3000.ost_server_uuid 50 [ 2750.187831] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8800a80c3000.ost_server_uuid in FULL state after 0 sec [ 2750.566724] Lustre: Permanently deactivating lustre-OST0001 [ 2750.574836] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 2750.578660] Lustre: Skipped 1 previous similar message [ 2751.266253] Lustre: Permanently reactivating lustre-OST0001 [ 2754.107665] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2754.109686] LustreError: Skipped 3 previous similar messages [ 2754.110732] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2754.113940] Lustre: Skipped 6 previous similar messages [ 2754.116275] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2754.118766] Lustre: Skipped 15 previous similar messages [ 2758.201990] Lustre: server umount lustre-OST0001 complete [ 2765.332887] Lustre: server umount lustre-OST0000 complete [ 2772.360624] Lustre: server umount lustre-MDT0000 complete [ 2773.671756] LustreError: 13247:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763767 with bad export cookie 17494770774756748435 [ 2773.672907] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2773.676468] LustreError: 13247:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2773.783493] Lustre: server umount lustre-MDT0001 complete [ 2775.369564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2776.397570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2777.601251] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2777.606465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2778.819958] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2778.826697] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2782.315513] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:42:55 (1712763775) [ 2782.634473] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2782.641129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2784.658394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2784.742220] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2784.751717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2784.754629] Lustre: Skipped 5 previous similar messages [ 2784.810500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2784.812859] Lustre: Skipped 8 previous similar messages [ 2785.706532] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2787.862336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2787.886572] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2788.896905] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2790.402144] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2791.191749] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2793.672838] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2793.678725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2793.703446] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2793.709445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2793.801468] Lustre: Permanently deactivating lustre-OST0000 [ 2795.681004] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2797.231109] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2799.235747] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2799.489563] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2799.495575] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2801.068734] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2802.616874] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2808.854395] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 2813.255771] Lustre: Permanently reactivating lustre-OST0000 [ 2817.163388] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 2817.167918] LustreError: 167-0: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2817.174471] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 2817.188265] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 2817.195170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 2824.868297] Lustre: server umount lustre-OST0001 complete [ 2828.893340] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2828.903637] LustreError: Skipped 9 previous similar messages [ 2832.403821] Lustre: server umount lustre-OST0000 complete [ 2839.821645] Lustre: server umount lustre-MDT0000 complete [ 2841.265635] LustreError: 20092:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763834 with bad export cookie 17494770774756750038 [ 2841.268441] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2841.280409] LustreError: 20092:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2841.450904] Lustre: server umount lustre-MDT0001 complete [ 2844.916499] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 2845.672120] Key type lgssc unregistered [ 2845.759518] LNet: 25182:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2845.764586] LNet: Removed LNI 192.168.202.152@tcp [ 2854.504479] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2854.617706] alg: No test for adler32 (adler32-zlib) [ 2855.368149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2855.424289] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 2855.469321] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 2855.471409] LNet: Accept secure, port 988 [ 2856.998376] Key type lgssc registered [ 2857.275868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2858.198627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2860.685210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2860.692556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.826887] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2861.838545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2862.931390] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2865.996513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2866.023473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2866.070950] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2866.084789] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2866.088844] Lustre: Permanently deactivating lustre-MDT0001 [ 2866.204989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2867.296093] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2868.749725] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2869.484518] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2871.187210] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 2871.189158] LustreError: 27179:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2871.194124] LustreError: 27179:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2871.198359] LustreError: 27179:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 2871.656717] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2871.662474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2871.799751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2872.798124] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2872.800672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 2873.451955] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2874.762679] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2875.798262] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2875.802199] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2875.814900] LustreError: Skipped 1 previous similar message [ 2877.180402] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2877.186271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2877.268034] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2878.554907] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 2878.563139] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 2878.933777] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2880.517572] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2881.943566] Lustre: Permanently reactivating lustre-MDT0001 [ 2881.947480] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 2881.949051] Lustre: Skipped 3 previous similar messages [ 2887.145136] Lustre: lustre-MDT0001: Client 7d2300fc-1b0a-47b3-a2a8-a4e4622da5fc (at 192.168.202.52@tcp) reconnecting [ 2888.667348] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2889.042311] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2890.037529] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 2890.097401] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 2890.909954] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 2890.956891] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 2891.590131] Lustre: Permanently deactivating lustre-MDT0001 [ 2891.595435] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 2891.598497] Lustre: Skipped 2 previous similar messages [ 2891.713946] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 2901.227749] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2901.230835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2901.234531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2904.615238] Lustre: server umount lustre-MDT0000 complete [ 2905.598679] LustreError: 27132:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712763898 with bad export cookie 3642265468739351951 [ 2905.599781] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2905.603424] LustreError: 27132:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2905.711181] Lustre: server umount lustre-MDT0001 complete [ 2912.733484] Lustre: server umount lustre-OST0000 complete [ 2917.391252] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:45:10 (1712763910) [ 2921.105539] Lustre: server umount lustre-OST0001 complete [ 2923.063908] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 2925.232277] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2928.094175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2930.018068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2931.934420] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2931.940947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2933.881864] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2933.891452] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2936.249104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2936.277169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2936.366871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2936.371975] Lustre: Skipped 2 previous similar messages [ 2936.382961] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2936.420524] Lustre: lustre-MDT0000: new disk, initializing [ 2936.448200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2936.455836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2937.386887] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2940.819616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2940.836766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2940.859428] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2940.871326] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2940.873061] Lustre: Skipped 1 previous similar message [ 2940.904492] Lustre: lustre-MDT0001: new disk, initializing [ 2940.935836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2940.940318] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2941.702808] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2943.965084] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2944.523755] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2946.363066] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2946.368468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2946.388967] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2946.391932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2946.465512] Lustre: lustre-OST0000: new disk, initializing [ 2946.467481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2946.488700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2946.492151] Lustre: Skipped 1 previous similar message [ 2947.789873] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2950.033446] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2950.310463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2950.316079] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2950.331464] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2955.550641] LustreError: 4247:0:(mdt_handler.c:3080:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 2959.410704] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2959.416317] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2959.432750] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2959.435984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2959.481819] Lustre: lustre-OST0001: new disk, initializing [ 2959.483807] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2960.813745] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2963.499666] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2965.558547] LustreError: 4247:0:(mdt_handler.c:3080:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 2965.566199] LustreError: 4247:0:(mdt_handler.c:3080:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 2965.954737] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 2969.298552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2969.302512] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2969.316112] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2975.571174] LustreError: 4247:0:(mdt_handler.c:3080:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 2979.324591] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2979.324740] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2979.325295] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2979.325296] Lustre: Skipped 3 previous similar messages [ 2979.334781] Lustre: Skipped 4 previous similar messages [ 2982.059979] Lustre: server umount lustre-OST0001 complete [ 2984.332231] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2984.336314] LustreError: Skipped 1 previous similar message [ 2989.340378] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2989.345177] LustreError: Skipped 1 previous similar message [ 2994.348264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2994.352748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2994.355299] Lustre: Skipped 3 previous similar messages [ 2999.239842] Lustre: server umount lustre-OST0000 complete [ 3001.019746] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3001.022943] LustreError: Skipped 1 previous similar message [ 3001.024547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3001.029397] Lustre: Skipped 1 previous similar message [ 3001.033280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3006.305621] Lustre: server umount lustre-MDT0000 complete [ 3007.682654] LustreError: 5191:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764001 with bad export cookie 3642265468739354716 [ 3007.684584] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3007.694417] LustreError: 5191:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3013.855395] Lustre: server umount lustre-MDT0001 complete [ 3017.725388] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3018.473283] Key type lgssc unregistered [ 3018.563756] LNet: 9665:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3018.567510] LNet: Removed LNI 192.168.202.152@tcp [ 3031.281517] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 3037.071989] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3037.282127] alg: No test for adler32 (adler32-zlib) [ 3038.033159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3038.149183] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 3038.216536] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3038.218657] LNet: Accept secure, port 988 [ 3039.771242] Key type lgssc registered [ 3040.134273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3043.587446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3046.151441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3048.460821] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3048.469731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3050.786908] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3050.793564] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3053.379661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3053.409384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3053.419300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.527535] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3054.548879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3054.586217] Lustre: lustre-MDT0000: new disk, initializing [ 3054.630953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3054.638827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3055.850035] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3060.297259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3060.335672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3060.376670] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3060.402663] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3060.406406] Lustre: Skipped 1 previous similar message [ 3060.451388] Lustre: lustre-MDT0001: new disk, initializing [ 3060.496534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3060.516624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3060.523545] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3061.665549] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3064.533221] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3065.307562] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3067.778710] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3067.783398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3067.806868] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3067.811490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3067.910726] Lustre: lustre-OST0000: new disk, initializing [ 3067.915073] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3067.960997] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3069.497836] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3069.526446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3069.530289] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3069.557058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3072.044821] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3073.021490] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3073.084954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3073.998922] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3074.060845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3076.815667] Lustre: server umount lustre-OST0000 complete [ 3079.550375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3079.551817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3079.558112] Lustre: Skipped 1 previous similar message [ 3084.557093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3084.561341] Lustre: Skipped 2 previous similar messages [ 3088.254852] Lustre: server umount lustre-MDT0000 complete [ 3089.565520] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3089.571702] LustreError: Skipped 1 previous similar message [ 3089.701228] LustreError: 14457:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764083 with bad export cookie 9661187394400386129 [ 3089.702068] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3089.711882] LustreError: 14457:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3099.863325] Lustre: server umount lustre-MDT0001 complete [ 3105.366300] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 11:48:18 (1712764098) [ 3107.665166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3107.868364] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3107.909788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3109.046398] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3112.035964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3112.188227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3113.389987] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3114.822258] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3115.689311] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3118.054625] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3118.063830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3119.804365] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3121.125451] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3132.219832] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3132.220308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3132.220676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3135.413929] Lustre: server umount lustre-OST0000 complete [ 3136.838808] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3136.845995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3140.740285] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3140.750404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3140.869348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3140.872481] Lustre: Skipped 1 previous similar message [ 3142.296215] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3143.519788] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3151.864846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3151.866920] Lustre: Skipped 1 previous similar message [ 3152.450417] Lustre: server umount lustre-OST0000 complete [ 3156.876028] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3156.876501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3156.876502] Lustre: Skipped 1 previous similar message [ 3156.887013] Lustre: Skipped 2 previous similar messages [ 3159.684253] Lustre: server umount lustre-MDT0000 complete [ 3160.830542] LustreError: 21452:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764154 with bad export cookie 9661187394400386969 [ 3160.831513] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3160.838924] LustreError: 21452:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3160.960037] Lustre: server umount lustre-MDT0001 complete [ 3164.251664] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3164.762232] Key type lgssc unregistered [ 3164.832647] LNet: 25330:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3164.835306] LNet: Removed LNI 192.168.202.152@tcp [ 3174.010102] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3174.136948] alg: No test for adler32 (adler32-zlib) [ 3174.888230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3174.969607] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 3175.026360] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3175.028794] LNet: Accept secure, port 988 [ 3176.565178] Key type lgssc registered [ 3176.823284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3178.994586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3179.004090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3180.163756] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3180.180225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3181.103363] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3183.891768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3184.038243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3184.946710] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3186.297666] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3187.094227] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3189.302601] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3189.312535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3189.453833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3190.455088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3191.101449] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3192.461388] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3204.475823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3204.475909] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3204.476225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3204.485394] Lustre: Skipped 1 previous similar message [ 3209.484591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3209.487185] Lustre: Skipped 2 previous similar messages [ 3209.952009] Lustre: server umount lustre-OST0000 complete [ 3214.059797] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3214.062363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3214.067222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3217.168437] Lustre: server umount lustre-MDT0000 complete [ 3218.450453] LustreError: 29079:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764211 with bad export cookie 17816899415431779755 [ 3218.451854] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3218.459458] LustreError: 29079:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3218.581940] Lustre: server umount lustre-MDT0001 complete [ 3222.092902] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3222.773860] Key type lgssc unregistered [ 3222.857944] LNet: 31880:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3222.862742] LNet: Removed LNI 192.168.202.152@tcp [ 3228.615317] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3228.796165] alg: No test for adler32 (adler32-zlib) [ 3229.548187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3229.627642] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 3229.695805] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3229.698670] LNet: Accept secure, port 988 [ 3231.244385] Key type lgssc registered [ 3231.521833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3233.840736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3233.848238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3235.017986] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3235.031603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3235.926255] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3238.539449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3238.665342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3239.632754] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3241.061037] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3241.868292] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3244.078020] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3244.083655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3244.216458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3245.216689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3245.586371] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3246.926444] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3255.228934] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3255.232626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3255.238408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3259.228106] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3259.228132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3259.228179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3260.680599] Lustre: server umount lustre-OST0000 complete [ 3263.675767] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3263.678608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3263.683665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3263.686365] Lustre: Skipped 1 previous similar message [ 3267.796617] Lustre: server umount lustre-MDT0000 complete [ 3268.802886] LustreError: 652:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764262 with bad export cookie 5525089277784859425 [ 3268.804437] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3268.814072] LustreError: 652:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3268.907137] Lustre: server umount lustre-MDT0001 complete [ 3272.039368] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3272.593447] Key type lgssc unregistered [ 3272.662439] LNet: 4555:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3272.665161] LNet: Removed LNI 192.168.202.152@tcp [ 3280.553616] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3280.667303] alg: No test for adler32 (adler32-zlib) [ 3281.419153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3281.476383] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 3281.520231] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3281.521949] LNet: Accept secure, port 988 [ 3283.045196] Key type lgssc registered [ 3283.255729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3285.039721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3285.043893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3286.156403] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3286.169556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3286.879000] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3288.974444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3289.058890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3289.735402] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3290.721974] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3291.285322] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3292.992300] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3292.997755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3293.096082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3294.392323] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3295.473177] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3296.095874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3308.493292] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3308.496075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3308.501102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3309.115713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3309.115883] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3309.115893] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3309.123189] Lustre: Skipped 1 previous similar message [ 3313.498488] Lustre: server umount lustre-OST0000 complete [ 3319.099963] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3319.102980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3319.106979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3320.615923] Lustre: server umount lustre-MDT0000 complete [ 3321.647855] LustreError: 6437:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764315 with bad export cookie 14269041812263368330 [ 3321.649059] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3321.654724] LustreError: 6437:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3321.753376] Lustre: server umount lustre-MDT0001 complete [ 3324.781453] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3325.264603] Key type lgssc unregistered [ 3325.330595] LNet: 11108:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3325.332548] LNet: Removed LNI 192.168.202.152@tcp [ 3330.284631] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3330.440617] alg: No test for adler32 (adler32-zlib) [ 3331.192157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3331.249463] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 3331.291914] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3331.293493] LNet: Accept secure, port 988 [ 3332.824202] Key type lgssc registered [ 3333.008956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3334.898246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3334.903088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3336.026443] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3336.037683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3336.770625] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3338.890688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3338.975295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3339.684087] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3340.788220] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3341.448176] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3343.240560] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3343.245297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3343.336810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3344.533267] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3345.588756] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3346.335620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3356.347619] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3356.349818] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3356.353021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3359.355642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3359.356098] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3359.356109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3359.368221] Lustre: Skipped 1 previous similar message [ 3360.439485] Lustre: server umount lustre-OST0000 complete [ 3363.995930] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3363.999196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3364.004127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3367.558112] Lustre: server umount lustre-MDT0000 complete [ 3368.649835] LustreError: 15411:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764362 with bad export cookie 12607332955666575152 [ 3368.651445] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3368.657343] LustreError: 15411:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3368.762115] Lustre: server umount lustre-MDT0001 complete [ 3371.773185] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3372.237724] Key type lgssc unregistered [ 3372.302710] LNet: 16206:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3372.305026] LNet: Removed LNI 192.168.202.152@tcp [ 3376.687813] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 3380.840494] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 3385.220911] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3385.365668] alg: No test for adler32 (adler32-zlib) [ 3386.117202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3386.178525] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 3386.226774] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3386.228926] LNet: Accept secure, port 988 [ 3387.755202] Key type lgssc registered [ 3387.958871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3389.935135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3391.606765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3393.098633] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3393.101725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3394.626290] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3394.629234] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3396.354600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3396.371055] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3396.374706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3397.439894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3397.450834] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3397.488692] Lustre: lustre-MDT0000: new disk, initializing [ 3397.507013] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3397.513678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3398.262592] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3401.690883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3401.707101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3401.733192] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3401.745811] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3401.747435] Lustre: Skipped 1 previous similar message [ 3401.778185] Lustre: lustre-MDT0001: new disk, initializing [ 3401.797419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3401.806919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3401.811253] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3402.600727] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3404.896126] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3405.529311] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3407.347043] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3407.352063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3407.375345] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3407.379511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3407.457706] Lustre: lustre-OST0000: new disk, initializing [ 3407.459849] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3407.480889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3408.797884] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3411.185351] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3412.104265] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3415.534408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3415.537898] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3415.556773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3416.224489] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3417.057804] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3417.108577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3420.555884] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3420.556112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3420.556394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3420.563226] LustreError: Skipped 1 previous similar message [ 3423.672720] Lustre: server umount lustre-OST0000 complete [ 3425.563724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3425.564013] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3425.564014] Lustre: Skipped 1 previous similar message [ 3425.568580] Lustre: Skipped 2 previous similar messages [ 3430.571518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3430.573055] Lustre: Skipped 2 previous similar messages [ 3430.638539] Lustre: server umount lustre-MDT0000 complete [ 3431.615760] LustreError: 21690:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764425 with bad export cookie 2751194333336228162 [ 3431.617092] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3431.626709] LustreError: 21690:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3431.725503] Lustre: server umount lustre-MDT0001 complete [ 3435.494650] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:53:49 (1712764429) [ 3437.262782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3437.398397] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3437.413441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3438.245929] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3440.609884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3441.556861] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3442.713277] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3443.356298] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3445.227563] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3445.231918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3445.328388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3445.330432] Lustre: Skipped 1 previous similar message [ 3446.620088] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3447.785237] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3456.331574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3456.332234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3456.332310] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3456.338302] Lustre: Skipped 1 previous similar message [ 3456.802253] Lustre: server umount lustre-OST0000 complete [ 3459.500336] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3459.502760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3459.740476] Lustre: server umount lustre-MDT0000 complete [ 3460.659704] LustreError: 28347:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764454 with bad export cookie 2751194333336229002 [ 3460.661444] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3460.661611] LustreError: 26770:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 3460.661784] LustreError: 26770:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -5 [ 3460.661787] LustreError: 26770:0:(llog_cat.c:773:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -5 [ 3460.676666] LustreError: 28347:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3460.755575] Lustre: server umount lustre-MDT0001 complete [ 3463.467489] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3463.890202] Key type lgssc unregistered [ 3463.950403] LNet: 29516:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3463.952358] LNet: Removed LNI 192.168.202.152@tcp [ 3471.225552] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 3475.326821] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3475.473090] alg: No test for adler32 (adler32-zlib) [ 3476.225140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3476.285497] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 3476.332113] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3476.333697] LNet: Accept secure, port 988 [ 3477.864228] Key type lgssc registered [ 3478.073946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3480.215354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3481.923123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3483.502516] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3483.505313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3485.079362] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3485.083433] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3486.324084] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3486.327313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3487.579400] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3487.582712] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3489.341104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3489.358627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3489.362701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3490.431257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3490.439684] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3490.475976] Lustre: lustre-MDT0000: new disk, initializing [ 3490.494547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3490.500733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3491.336385] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3494.927703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3494.945341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3494.972577] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3494.983611] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3494.985256] Lustre: Skipped 1 previous similar message [ 3495.018663] Lustre: lustre-MDT0001: new disk, initializing [ 3495.041536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3495.051645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3495.055385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3495.857786] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3498.258381] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3498.941076] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3500.637270] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3500.640299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3500.658414] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3500.661536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3500.728695] Lustre: lustre-OST2710: new disk, initializing [ 3500.730537] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 3500.746877] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 3501.957841] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3503.190973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 3503.193550] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 3503.210520] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 3504.297729] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 3505.960497] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3505.963069] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3505.981450] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3505.984262] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3506.016721] Lustre: lustre-OST03e8: new disk, initializing [ 3506.018767] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 3506.034839] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 3507.195632] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3508.210665] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 3510.170015] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 3510.212865] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3510.637456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 3510.641105] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 3510.652485] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 3510.939143] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 3510.987213] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3511.760746] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 3511.816396] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3512.547673] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 3512.590745] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3515.067658] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3515.070584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3515.075301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3515.660072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3515.660439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3515.667714] Lustre: Skipped 1 previous similar message [ 3519.359241] Lustre: server umount lustre-MDT0000 complete [ 3520.278005] LustreError: 2064:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764513 with bad export cookie 15145497326958606432 [ 3520.278947] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3520.285837] LustreError: 2064:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3520.406460] Lustre: server umount lustre-MDT0001 complete [ 3531.411096] Lustre: server umount lustre-OST2710 complete [ 3542.522991] Lustre: server umount lustre-OST03e8 complete [ 3546.790395] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 3548.384697] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3550.611528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3552.194874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3553.653795] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3553.656789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3555.119859] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3555.123424] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3556.820105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3556.836320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3556.899767] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3556.907923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3556.938744] Lustre: lustre-MDT0000: new disk, initializing [ 3556.955380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3556.960455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3557.654625] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3560.940151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3560.962255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3560.983816] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3561.048699] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3561.780410] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3563.986278] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3564.573989] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3566.254813] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3566.257545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3566.276629] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3566.279746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3566.346985] Lustre: lustre-OST0000: new disk, initializing [ 3566.348032] Lustre: Skipped 1 previous similar message [ 3566.349600] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3566.350938] Lustre: Skipped 2 previous similar messages [ 3566.366584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3566.368760] Lustre: Skipped 1 previous similar message [ 3567.549835] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3569.740452] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3570.460754] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3572.489859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3572.494397] Lustre: Skipped 1 previous similar message [ 3572.496506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3572.511184] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3573.552898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3574.389341] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3574.447308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3577.516017] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3577.516086] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3577.516529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3577.516531] Lustre: Skipped 2 previous similar messages [ 3577.525419] Lustre: Skipped 2 previous similar messages [ 3581.027799] Lustre: server umount lustre-OST0000 complete [ 3582.523850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3582.529420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3582.531456] Lustre: Skipped 2 previous similar messages [ 3587.531807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3587.535789] Lustre: Skipped 1 previous similar message [ 3588.122702] Lustre: server umount lustre-MDT0000 complete [ 3589.144737] LustreError: 11211:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764582 with bad export cookie 15145497326958608070 [ 3589.145924] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3589.151111] LustreError: 11211:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3589.249858] Lustre: server umount lustre-MDT0001 complete [ 3592.929631] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:56:26 (1712764586) [ 3593.270600] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 3595.281460] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:56:28 (1712764588) [ 3595.792488] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3597.472666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3598.481378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3599.496249] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3599.499825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3600.482578] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3600.485619] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3600.758317] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3600.762542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3602.417205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3602.491367] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3602.497198] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3602.542761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3603.225811] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3605.292187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3605.314142] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3606.060254] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3607.052917] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3607.636327] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3609.258336] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3609.261577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3609.282377] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3609.286804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3609.349599] Lustre: Denying initial registration attempt from nid 192.168.202.152@tcp, specified as failover [ 3609.352089] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 3609.355422] LustreError: 20352:0:(tgt_mount.c:2223:server_fill_super()) Unable to start targets: -99 [ 3609.358265] LustreError: 20352:0:(tgt_mount.c:1755:server_put_super()) no obd lustre-OST0000 [ 3609.360020] LustreError: 20352:0:(tgt_mount.c:135:server_deregister_mount()) lustre-OST0000 not registered [ 3609.406781] Lustre: server umount lustre-OST0000 complete [ 3609.407860] LustreError: 20352:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 3615.387683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3615.387738] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3615.387739] LustreError: Skipped 1 previous similar message [ 3615.388078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3615.395596] Lustre: Skipped 2 previous similar messages [ 3617.190745] Lustre: server umount lustre-MDT0000 complete [ 3618.182773] LustreError: 18497:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764611 with bad export cookie 15145497326958608910 [ 3618.183943] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3618.191007] LustreError: 18497:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3618.292898] Lustre: server umount lustre-MDT0001 complete [ 3621.341178] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3621.876737] Key type lgssc unregistered [ 3621.947532] LNet: 21628:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3621.951131] LNet: Removed LNI 192.168.202.152@tcp [ 3626.206748] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3626.336615] alg: No test for adler32 (adler32-zlib) [ 3627.088158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3627.145143] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 3627.194648] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3627.196324] LNet: Accept secure, port 988 [ 3628.724239] Key type lgssc registered [ 3628.930850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3630.683900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3631.707586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3632.748361] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3632.753288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3633.833924] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3633.837750] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3634.202278] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3634.207001] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3638.772255] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3640.724093] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3640.730060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3641.811560] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3641.819598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3641.871375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3642.655315] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3644.908482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3644.936216] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3644.944884] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3645.019295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3645.801012] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3646.842339] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3647.428816] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3649.156308] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3649.159460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3649.175329] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3649.179533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3649.236597] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3649.271771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3650.480510] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3651.534863] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3654.395629] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3654.398648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3654.403402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3658.110094] Lustre: server umount lustre-OST0000 complete [ 3659.419794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3659.420137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3659.420138] Lustre: Skipped 1 previous similar message [ 3659.426551] Lustre: Skipped 2 previous similar messages [ 3664.427559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3664.429636] Lustre: Skipped 2 previous similar messages [ 3665.157357] Lustre: server umount lustre-MDT0000 complete [ 3666.146833] LustreError: 25148:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764659 with bad export cookie 8255550118485402555 [ 3666.148066] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3666.154949] LustreError: 25148:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3666.253584] Lustre: server umount lustre-MDT0001 complete [ 3668.133823] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3668.612205] Key type lgssc unregistered [ 3668.678369] LNet: 28806:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3668.681134] LNet: Removed LNI 192.168.202.152@tcp [ 3676.718145] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3676.869085] alg: No test for adler32 (adler32-zlib) [ 3677.620241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3677.682828] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 3677.730597] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3677.732477] LNet: Accept secure, port 988 [ 3679.260175] Key type lgssc registered [ 3679.460234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3681.350239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3681.355297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3682.477017] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3682.488276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3683.237217] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3685.469304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3685.566065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3686.338757] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3687.390737] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3687.970891] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3689.710404] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3689.714914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3689.802115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3691.011726] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3692.047209] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3699.819829] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3699.820184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3699.825169] Lustre: Skipped 2 previous similar messages [ 3703.163372] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 3703.165091] Lustre: Skipped 2 previous similar messages [ 3704.827713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3704.830274] Lustre: Skipped 2 previous similar messages [ 3705.352744] Lustre: server umount lustre-MDT0000 complete [ 3706.308919] LustreError: 31570:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764699 with bad export cookie 8363064911004921707 [ 3706.310122] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3706.318586] LustreError: 31570:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3708.165204] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3708.174434] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.52@tcp (stopping) [ 3709.835623] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3709.835794] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3712.413490] Lustre: server umount lustre-MDT0001 complete [ 3714.072165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3716.282521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3730.837399] LustreError: 29938:0:(client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8800a1698e00 x1795963927668608/t0(0) o250->MGC192.168.202.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 3730.929411] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3730.933693] LustreError: Skipped 1 previous similar message [ 3730.943240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3730.954136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 3731.685278] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3733.897875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3734.734565] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3735.776827] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3735.990461] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 3739.204346] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3739.748898] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3740.994189] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 3749.258113] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 9 sec [ 3750.971598] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3750.973760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3750.977688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3750.981076] Lustre: Skipped 2 previous similar messages [ 3756.207694] Lustre: server umount lustre-OST0000 complete [ 3759.019605] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3759.023645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3759.029730] Lustre: Skipped 1 previous similar message [ 3759.032700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3759.034992] Lustre: Skipped 3 previous similar messages [ 3763.260366] Lustre: server umount lustre-MDT0000 complete [ 3764.279010] LustreError: 30682:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764757 with bad export cookie 8363064911004928833 [ 3764.280650] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3764.289390] LustreError: 30682:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3764.406102] Lustre: server umount lustre-MDT0001 complete [ 3767.263196] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3767.774454] Key type lgssc unregistered [ 3767.838362] LNet: 4971:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3767.841321] LNet: Removed LNI 192.168.202.152@tcp [ 3775.672923] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3775.826287] alg: No test for adler32 (adler32-zlib) [ 3776.577117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3776.636861] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 3776.681247] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3776.683499] LNet: Accept secure, port 988 [ 3778.211235] Key type lgssc registered [ 3778.404028] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3780.233582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3780.238292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3781.357535] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3781.367455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3782.093084] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3784.264855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3784.360307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3785.091903] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3786.110693] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3786.705337] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3787.486218] LustreError: 6896:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3787.489323] LustreError: 6896:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3787.491695] LustreError: 6896:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 3789.355845] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3789.358391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3789.362495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3793.548836] Lustre: server umount lustre-MDT0000 complete [ 3794.372719] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3794.377705] LustreError: Skipped 1 previous similar message [ 3794.506908] LustreError: 7738:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764787 with bad export cookie 11193924118389138749 [ 3794.508124] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3794.513148] LustreError: 7738:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3794.606780] Lustre: server umount lustre-MDT0001 complete [ 3798.290400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3798.368628] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3798.377208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3798.433021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3799.158510] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3801.314124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3801.338356] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3801.347197] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3801.411650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3802.133922] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3803.157936] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3803.715106] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3805.661279] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3805.665595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3805.722187] LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 3805.725135] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 3805.727962] LustreError: 11632:0:(tgt_mount.c:2223:server_fill_super()) Unable to start targets: -2 [ 3805.729732] LustreError: 11632:0:(tgt_mount.c:1755:server_put_super()) no obd lustre-OST0000 [ 3805.731585] LustreError: 11632:0:(tgt_mount.c:135:server_deregister_mount()) lustre-OST0000 not registered [ 3805.775553] Lustre: server umount lustre-OST0000 complete [ 3805.776934] LustreError: 11632:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3807.917342] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3807.920611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3807.976968] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3808.013004] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3809.234776] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3810.301440] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3812.243914] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3812.247642] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3812.264190] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3812.267256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3812.286319] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3812.302489] Lustre: lustre-OST0001: new disk, initializing [ 3812.304423] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3813.548102] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3814.608335] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3815.381356] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3815.389090] Lustre: server umount lustre-OST0001 complete [ 3815.401932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 3820.395705] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3820.398237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3820.401264] Lustre: Skipped 1 previous similar message [ 3820.403363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3820.404822] Lustre: Skipped 1 previous similar message [ 3820.411846] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3820.416255] LustreError: Skipped 2 previous similar messages [ 3822.431526] Lustre: server umount lustre-OST0000 complete [ 3825.419604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3825.420127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3825.420129] Lustre: Skipped 1 previous similar message [ 3825.425286] Lustre: Skipped 2 previous similar messages [ 3829.450229] Lustre: server umount lustre-MDT0000 complete [ 3830.427474] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3830.430688] LustreError: Skipped 1 previous similar message [ 3830.435766] LustreError: 9710:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764823 with bad export cookie 11193924118389139267 [ 3830.436998] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3830.444941] LustreError: 9710:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3830.540417] Lustre: server umount lustre-MDT0001 complete [ 3832.384785] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3832.901360] Key type lgssc unregistered [ 3832.973447] LNet: 15467:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3832.976540] LNet: Removed LNI 192.168.202.152@tcp [ 3841.240725] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 3845.588929] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3845.750411] alg: No test for adler32 (adler32-zlib) [ 3846.502115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3846.571546] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 3846.616300] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3846.617639] LNet: Accept secure, port 988 [ 3848.142236] Key type lgssc registered [ 3848.348206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3850.530451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3852.166831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3853.653356] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3853.657400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3855.197688] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3855.200619] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3856.932109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3856.945356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3856.949521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3858.015771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3858.024309] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3858.057487] Lustre: lustre-MDT0000: new disk, initializing [ 3858.075121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3858.079828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3858.823806] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3862.243437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3862.262958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3862.286452] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3862.299030] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3862.300681] Lustre: Skipped 1 previous similar message [ 3862.333682] Lustre: lustre-MDT0001: new disk, initializing [ 3862.354370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3862.364120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3862.369462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3863.136217] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3865.405389] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3865.987168] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3867.691531] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3867.694312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3867.710675] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3867.713371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3867.782123] Lustre: lustre-OST0000: new disk, initializing [ 3867.784354] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3867.801571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3869.019243] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3870.556151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3870.559612] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3870.574758] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3871.266381] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3872.076483] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3872.121549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3872.912658] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3872.962471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3875.579912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3875.580312] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3875.580364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3875.586057] Lustre: Skipped 1 previous similar message [ 3879.506065] Lustre: server umount lustre-OST0000 complete [ 3880.588091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3880.588701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3880.588703] Lustre: Skipped 1 previous similar message [ 3880.603897] Lustre: Skipped 1 previous similar message [ 3885.595523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3885.597265] Lustre: Skipped 2 previous similar messages [ 3886.577468] Lustre: server umount lustre-MDT0000 complete [ 3887.545758] LustreError: 20248:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764880 with bad export cookie 7871709413062922112 [ 3887.546999] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3887.552741] LustreError: 20248:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3887.665888] Lustre: server umount lustre-MDT0001 complete [ 3891.250590] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 12:01:24 (1712764884) [ 3892.807731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3894.444674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3899.919543] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 3901.492970] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3903.730077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3905.361565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3906.856951] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3906.859958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3908.425456] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3908.428202] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3910.266455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3910.283744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3910.349505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3910.358202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3910.394241] Lustre: lustre-MDT0000: new disk, initializing [ 3910.411456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3910.416516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3911.117794] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3914.406392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3914.423580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3914.449355] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3914.461326] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3914.462961] Lustre: Skipped 1 previous similar message [ 3914.495328] Lustre: lustre-MDT0001: new disk, initializing [ 3914.514990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3914.522992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3914.525379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3915.257438] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3917.495892] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3918.073562] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3919.770578] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3919.773859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3919.787430] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3919.789912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3921.046444] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3923.275337] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3924.074098] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3924.792787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3924.796812] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3924.810975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3925.138638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3925.919685] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3925.963870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3929.804384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3929.809377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3929.812365] Lustre: Skipped 1 previous similar message [ 3932.535196] Lustre: server umount lustre-OST0000 complete [ 3934.539654] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3934.542069] LustreError: Skipped 1 previous similar message [ 3934.543354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3934.547281] Lustre: Skipped 1 previous similar message [ 3934.549293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3939.580924] Lustre: server umount lustre-MDT0000 complete [ 3939.819553] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3939.825815] LustreError: Skipped 1 previous similar message [ 3940.542823] LustreError: 29209:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712764933 with bad export cookie 7871709413062922952 [ 3940.544097] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3940.552553] LustreError: 29209:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3940.651151] Lustre: server umount lustre-MDT0001 complete [ 3944.264176] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:02:17 (1712764937) [ 3947.578436] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:02:21 (1712764941) [ 3949.748467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3949.879270] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3949.894065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3949.897083] Lustre: Skipped 1 previous similar message [ 3950.649081] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3952.795498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3953.622892] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3954.670958] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3955.228729] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3956.923773] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3956.926988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3958.190229] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3959.218247] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3968.421273] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 3969.021634] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 3969.554290] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 3970.027371] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 3970.667974] Lustre: Failing over lustre-MDT0000 [ 3970.757593] Lustre: server umount lustre-MDT0000 complete [ 3972.907717] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3972.911834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3972.917670] Lustre: Skipped 1 previous similar message [ 3972.920670] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3975.354064] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3975.357326] LustreError: Skipped 2 previous similar messages [ 3980.357500] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3980.367246] LustreError: Skipped 3 previous similar messages [ 3982.877087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3982.905889] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3982.973932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3982.975466] Lustre: Skipped 2 previous similar messages [ 3982.984434] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3983.694041] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3985.365468] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3987.981904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 3987.986731] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3988.001997] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3988.587087] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3988.955181] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3990.640005] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 3991.137561] Lustre: Failing over lustre-MDT0000 [ 3991.198030] Lustre: server umount lustre-MDT0000 complete [ 3992.988401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3992.988869] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3992.998176] Lustre: Skipped 4 previous similar messages [ 4003.167984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4003.199096] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4003.203447] LustreError: 7385:0:(ldlm_resource.c:1128:ldlm_resource_complain()) MGC192.168.202.152@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff880087346400) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4003.210243] LustreError: 2749:0:(mgc_request.c:621:do_requeue()) failed processing log: -5 [ 4003.282622] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4003.983202] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4005.397355] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4008.286335] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 4008.289538] Lustre: Skipped 2 previous similar messages [ 4008.297641] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4008.322605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4009.008017] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4009.446465] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4013.291676] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4013.292094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4013.292096] Lustre: Skipped 2 previous similar messages [ 4013.297169] Lustre: Skipped 2 previous similar messages [ 4017.426558] Lustre: server umount lustre-MDT0000 complete [ 4018.299703] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4018.303345] LustreError: Skipped 12 previous similar messages [ 4018.376915] LustreError: 2737:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712765011 with bad export cookie 7871709413062925388 [ 4018.378538] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4018.386673] LustreError: 2737:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4018.483828] Lustre: server umount lustre-MDT0001 complete [ 4029.620294] Lustre: server umount lustre-OST0000 complete [ 4033.430029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4033.580133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4033.582316] Lustre: Skipped 1 previous similar message [ 4034.299385] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4036.452939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4037.286410] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4038.303462] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4038.887234] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4040.560601] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4040.565934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4041.826725] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4042.659531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4042.849692] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4050.668137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4050.668195] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4050.668550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4050.668551] Lustre: Skipped 2 previous similar messages [ 4050.679205] Lustre: Skipped 1 previous similar message [ 4054.694024] Lustre: server umount lustre-OST0000 complete [ 4061.708361] Lustre: server umount lustre-MDT0000 complete [ 4062.664714] LustreError: 9661:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712765056 with bad export cookie 7871709413062926116 [ 4062.665925] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4062.674899] LustreError: 9661:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4062.772168] Lustre: server umount lustre-MDT0001 complete [ 4065.616925] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 4066.056547] Key type lgssc unregistered [ 4066.117316] LNet: 13300:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4066.120837] LNet: Removed LNI 192.168.202.152@tcp [ 4074.373265] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4074.526385] alg: No test for adler32 (adler32-zlib) [ 4075.277178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4075.335103] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 4075.379810] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 4075.381387] LNet: Accept secure, port 988 [ 4076.908196] Key type lgssc registered [ 4077.096081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4078.883775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4078.888130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4079.999375] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4080.009955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4080.727843] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4082.860848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4082.950108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4083.689762] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4084.693424] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4085.273007] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4086.921942] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4086.924824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4087.009301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4088.246564] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4089.249513] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4090.010805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 4102.971615] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4102.973663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4102.976920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4104.602023] Lustre: server umount lustre-MDT0000 complete [ 4105.565729] LustreError: 17880:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712765098 with bad export cookie 4560301361756618461 [ 4105.567152] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4105.570428] LustreError: 17880:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4105.670628] Lustre: server umount lustre-MDT0001 complete [ 4116.714851] Lustre: server umount lustre-OST0000 complete [ 4120.011026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4120.137940] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4120.155492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4120.884268] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4122.979465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4123.820856] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4124.847850] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4125.405738] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4127.072578] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4127.075349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4127.157163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4127.160284] Lustre: Skipped 1 previous similar message [ 4128.348350] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4129.328161] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4130.157601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4143.099625] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4143.101684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4143.104474] Lustre: Skipped 2 previous similar messages [ 4143.105966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4143.107592] Lustre: Skipped 2 previous similar messages [ 4146.640085] Lustre: server umount lustre-MDT0000 complete [ 4147.616771] LustreError: 19358:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712765141 with bad export cookie 4560301361756619532 [ 4147.617800] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4147.622305] LustreError: 19358:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4147.723815] Lustre: server umount lustre-MDT0001 complete [ 4158.776118] Lustre: server umount lustre-OST0000 complete [ 4162.645858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4162.770520] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4162.786205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4163.512292] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4165.645095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4166.452167] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4167.467927] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4168.028513] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4169.757888] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4169.762305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4171.038499] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4172.058368] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4172.850443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4180.859922] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4180.860491] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4180.860541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4180.860542] Lustre: Skipped 2 previous similar messages [ 4180.867994] Lustre: Skipped 3 previous similar messages [ 4183.945539] Lustre: server umount lustre-OST0000 complete [ 4185.755677] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4185.758006] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4185.761641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4185.763315] Lustre: Skipped 1 previous similar message [ 4190.875457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4190.877251] Lustre: Skipped 3 previous similar messages [ 4190.944623] Lustre: server umount lustre-MDT0000 complete [ 4191.895675] LustreError: 26517:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712765185 with bad export cookie 4560301361756620512 [ 4191.896857] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4191.903845] LustreError: 26517:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4191.996235] Lustre: server umount lustre-MDT0001 complete [ 4194.887635] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 4195.432832] Key type lgssc unregistered [ 4195.490405] LNet: 27312:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4195.493510] LNet: Removed LNI 192.168.202.152@tcp [ 4203.335550] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4203.491273] alg: No test for adler32 (adler32-zlib) [ 4204.242169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4204.300162] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 4204.350002] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 4204.352243] LNet: Accept secure, port 988 [ 4205.878300] Key type lgssc registered [ 4206.077489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4207.920413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4207.925479] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4207.928245] LustreError: 29248:0:(osd_handler.c:8471:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 4207.931116] LustreError: 29248:0:(obd_config.c:797:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 4207.934182] LustreError: 29248:0:(obd_mount.c:215:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 4207.938148] LustreError: 29248:0:(tgt_mount.c:2190:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 4207.942714] LustreError: 29248:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4209.835408] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4209.838661] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4209.841017] LustreError: 29772:0:(osd_handler.c:8471:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 4209.844671] LustreError: 29772:0:(obd_config.c:797:class_setup()) setup lustre-OST0000-osd failed (-22) [ 4209.846871] LustreError: 29772:0:(obd_mount.c:215:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 4209.849246] LustreError: 29772:0:(tgt_mount.c:2190:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 4209.852057] LustreError: 29772:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4213.821014] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 4214.277695] Key type lgssc unregistered [ 4214.342373] LNet: 30895:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4214.345050] LNet: Removed LNI 192.168.202.152@tcp [ 4218.747537] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 4223.406780] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4223.524262] alg: No test for adler32 (adler32-zlib) [ 4224.275138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4224.335580] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 4224.378166] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 4224.379808] LNet: Accept secure, port 988 [ 4225.903372] Key type lgssc registered [ 4226.116535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4228.178495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4229.784782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4231.275783] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4231.278413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4232.764910] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 4232.769009] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4234.468097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4234.480206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4234.483974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4235.553496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4235.562247] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4235.595819] Lustre: lustre-MDT0000: new disk, initializing [ 4235.617213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4235.622597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4236.343685] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4239.740180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4239.755719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4239.776399] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4239.786976] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4239.789534] Lustre: Skipped 1 previous similar message [ 4239.822623] Lustre: lustre-MDT0001: new disk, initializing [ 4239.842101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4239.850336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4239.852914] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4240.590521] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4242.825967] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4243.369031] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4245.050344] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4245.053984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4245.074634] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4245.077848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4245.145982] Lustre: lustre-OST0000: new disk, initializing [ 4245.147831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4245.167068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4246.339093] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4248.565795] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4249.347778] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4251.780525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4251.784395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4251.800905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4252.430848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4253.168543] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4253.214177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4256.795659] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4256.797843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4256.801136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4259.749860] Lustre: server umount lustre-OST0000 complete [ 4261.819644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4261.820140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4261.820142] Lustre: Skipped 1 previous similar message [ 4261.826297] Lustre: Skipped 2 previous similar messages [ 4266.738037] Lustre: server umount lustre-MDT0000 complete [ 4266.827547] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4266.830965] LustreError: Skipped 1 previous similar message [ 4267.798008] LustreError: 4210:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712765261 with bad export cookie 3331340863245049292 [ 4267.799572] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4267.808973] LustreError: 4210:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4267.900910] Lustre: server umount lustre-MDT0001 complete [ 4270.378325] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 4271.749546] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:07:45 (1712765265) [ 4273.331701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4273.468016] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4273.482778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4274.138720] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4276.264162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4277.060918] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4278.057030] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4278.602414] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4280.226366] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4280.228908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4280.309855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4280.311440] Lustre: Skipped 1 previous similar message [ 4281.456885] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4282.465705] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4284.180298] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 4284.183766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4284.201885] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 4284.204722] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4284.239114] Lustre: lustre-OST0001: new disk, initializing [ 4284.240686] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4285.443901] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4286.999185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4287.002091] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4287.018091] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4287.692924] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4292.011787] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4292.012846] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4292.013232] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4292.013233] Lustre: Skipped 1 previous similar message [ 4292.021646] LustreError: Skipped 2 previous similar messages [ 4295.318569] Lustre: server umount lustre-OST0001 complete [ 4297.020336] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4297.023559] LustreError: Skipped 1 previous similar message [ 4302.028285] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4302.031803] LustreError: Skipped 1 previous similar message [ 4307.036166] LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4307.036315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4307.036316] Lustre: Skipped 1 previous similar message [ 4307.039375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4307.039377] Lustre: Skipped 2 previous similar messages [ 4307.047193] LustreError: Skipped 1 previous similar message [ 4312.044029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4312.045759] Lustre: Skipped 2 previous similar messages [ 4312.498009] Lustre: server umount lustre-OST0000 complete [ 4316.411776] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4316.415299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4316.420487] Lustre: Skipped 1 previous similar message [ 4319.489228] Lustre: server umount lustre-MDT0000 complete [ 4320.460870] LustreError: 7379:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712765313 with bad export cookie 3331340863245050132 [ 4320.461992] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4320.467249] LustreError: 7379:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4326.589251] Lustre: server umount lustre-MDT0001 complete [ 4330.008661] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 4330.483851] Key type lgssc unregistered [ 4330.555442] LNet: 12421:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4330.557549] LNet: Removed LNI 192.168.202.152@tcp [ 4338.712651] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 4343.227380] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4343.415625] alg: No test for adler32 (adler32-zlib) [ 4344.167225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4344.233077] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 4344.283363] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 4344.285731] LNet: Accept secure, port 988 [ 4345.817235] Key type lgssc registered [ 4346.008416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4348.145264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4349.787554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4351.318992] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4351.322147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4352.934550] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 4352.938061] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4354.713355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4354.730681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4354.736161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4355.818658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4355.826477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4355.859778] Lustre: lustre-MDT0000: new disk, initializing [ 4355.879026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4355.885274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4356.730881] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4360.314377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4360.329719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4360.351221] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4360.364489] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4360.366709] Lustre: Skipped 1 previous similar message [ 4360.403684] Lustre: lustre-MDT0001: new disk, initializing [ 4360.426656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4360.435906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4360.439162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4361.200230] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4363.506036] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4364.087377] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4365.804074] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4365.807104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4365.825464] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4365.828967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4365.898228] Lustre: lustre-OST0000: new disk, initializing [ 4365.900182] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4365.918371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4367.130230] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4369.410158] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4370.232739] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4373.083859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4373.086867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4373.100879] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4373.337467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4374.164682] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4374.216124] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4378.107996] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4378.108078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4378.108341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4378.116255] LustreError: Skipped 1 previous similar message [ 4380.802550] Lustre: server umount lustre-OST0000 complete [ 4383.115620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4383.115879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4383.115880] Lustre: Skipped 1 previous similar message [ 4383.122619] Lustre: Skipped 2 previous similar messages [ 4387.842209] Lustre: server umount lustre-MDT0000 complete [ 4388.123525] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4388.126729] LustreError: Skipped 1 previous similar message [ 4388.806021] LustreError: 17203:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712765382 with bad export cookie 964426043915513529 [ 4388.807095] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4388.814925] LustreError: 17203:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4388.911277] Lustre: server umount lustre-MDT0001 complete [ 4392.499516] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 12:09:46 (1712765386) [ 4394.843689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4396.955840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4397.082814] Lustre: lustre-MDT0000: new disk, initializing [ 4397.093409] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4397.111637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4397.878791] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4400.092524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4400.947185] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4401.965543] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4402.517570] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4403.034152] LustreError: 23165:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4403.037088] LustreError: 23165:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4403.039974] LustreError: 23165:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 4405.179698] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4405.181913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4405.185517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4405.187460] Lustre: Skipped 1 previous similar message [ 4409.094616] Lustre: server umount lustre-MDT0000 complete [ 4410.060947] LustreError: 23120:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712765403 with bad export cookie 964426043915514369 [ 4410.062531] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4410.068235] LustreError: 23120:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4410.163094] Lustre: server umount lustre-MDT0001 complete [ 4413.893027] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:10:07 (1712765407) [ 4415.528816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4415.653766] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4415.668659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4415.670340] Lustre: Skipped 1 previous similar message [ 4416.378266] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4418.480021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4419.299641] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4420.339318] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4420.912471] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4422.602713] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4422.606519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4423.817107] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4424.838682] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4433.108222] Lustre: Permanently deactivating lustre-OST0000 [ 4433.112106] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [ 4434.974829] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4436.098232] LustreError: 29188:0:(mgs_llog.c:1608:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [ 4436.105451] LustreError: 29188:0:(mgs_llog.c:1662:mgs_replace_nids()) Only MGS is allowed to be started [ 4436.110905] LustreError: 29188:0:(mgs_handler.c:1099:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 4443.098006] Lustre: server umount lustre-OST0000 complete [ 4443.851058] LustreError: 29588:0:(mgs_llog.c:1662:mgs_replace_nids()) Only MGS is allowed to be started [ 4443.853667] LustreError: 29588:0:(mgs_llog.c:1662:mgs_replace_nids()) Skipped 1 previous similar message [ 4443.855960] LustreError: 29588:0:(mgs_handler.c:1099:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 4443.858332] LustreError: 29588:0:(mgs_handler.c:1099:mgs_iocontrol()) Skipped 1 previous similar message [ 4447.723770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4447.724114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4447.724116] Lustre: Skipped 1 previous similar message [ 4447.730141] Lustre: Skipped 2 previous similar messages [ 4452.732300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4452.736187] Lustre: Skipped 2 previous similar messages [ 4454.440828] Lustre: server umount lustre-MDT0000 complete [ 4455.522033] LustreError: 26221:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712765448 with bad export cookie 964426043915514901 [ 4455.524015] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4455.528871] LustreError: 26221:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4465.651219] Lustre: server umount lustre-MDT0001 complete [ 4467.863059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4468.623689] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4469.328534] LustreError: 31038:0:(mgs_handler.c:1099:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 4470.964787] Lustre: 31496:0:(mgs_llog.c:1369:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.152@tcp [ 4471.510636] Lustre: server umount MGS complete [ 4474.515533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4474.653140] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4474.656700] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4474.656702] Lustre: Skipped 1 previous similar message [ 4474.669593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4474.671309] Lustre: Skipped 2 previous similar messages [ 4475.395576] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4477.606365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4477.685902] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4478.445141] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4479.475295] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4480.036487] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4480.507195] Lustre: Permanently reactivating lustre-OST0000 [ 4480.510642] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 4480.512215] Lustre: Skipped 2 previous similar messages [ 4505.175120] LustreError: 1808:0:(osp_dev.c:696:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 4508.027473] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4508.030237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4508.118613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4508.120860] Lustre: Skipped 1 previous similar message [ 4509.286311] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4510.261971] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4525.132254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4525.132262] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4525.132546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4525.140656] Lustre: Skipped 1 previous similar message [ 4527.439427] Lustre: server umount lustre-OST0000 complete [ 4534.442675] Lustre: server umount lustre-MDT0000 complete [ 4535.147439] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4535.151059] LustreError: Skipped 1 previous similar message [ 4535.398797] LustreError: 32428:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712765528 with bad export cookie 964426043915515804 [ 4535.400094] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4535.404776] LustreError: 32428:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4535.502880] Lustre: server umount lustre-MDT0001 complete [ 4538.312416] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 4538.802725] Key type lgssc unregistered [ 4538.864513] LNet: 5228:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4538.868015] LNet: Removed LNI 192.168.202.152@tcp [ 4543.324771] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 4547.471792] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4547.596538] alg: No test for adler32 (adler32-zlib) [ 4548.348112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4548.406827] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 4548.453279] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 4548.454805] LNet: Accept secure, port 988 [ 4549.983191] Key type lgssc registered [ 4550.170761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4552.211245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4553.787796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4555.234767] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4555.237956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4556.711790] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 4556.714625] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4558.408029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4558.422275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4558.426489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4559.493018] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4559.500702] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4559.533489] Lustre: lustre-MDT0000: new disk, initializing [ 4559.550019] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4559.555748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4560.284208] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4563.647063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4563.662793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4563.686339] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4563.698728] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4563.700305] Lustre: Skipped 1 previous similar message [ 4563.737912] Lustre: lustre-MDT0001: new disk, initializing [ 4563.759116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4563.766911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4563.770365] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4564.537410] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4566.811529] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4567.425123] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4569.341525] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4569.344183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4569.362609] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4569.364992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4569.442505] Lustre: lustre-OST0000: new disk, initializing [ 4569.444117] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4569.460818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4570.678713] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4572.938466] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4573.360421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4573.363539] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4573.380265] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4573.709813] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4573.756206] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4574.487991] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4574.530690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4578.379857] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4578.380107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4578.380562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4578.389456] LustreError: Skipped 1 previous similar message [ 4581.077904] Lustre: server umount lustre-OST0000 complete [ 4583.387673] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4583.387959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4583.387961] Lustre: Skipped 1 previous similar message [ 4583.394196] Lustre: Skipped 2 previous similar messages [ 4588.085521] Lustre: server umount lustre-MDT0000 complete [ 4588.395544] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4588.399555] LustreError: Skipped 1 previous similar message [ 4589.048815] LustreError: 8936:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712765582 with bad export cookie 18143196286468897120 [ 4589.050081] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4589.054795] LustreError: 8936:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4589.151649] Lustre: server umount lustre-MDT0001 complete [ 4591.629033] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 4593.017755] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:13:06 (1712765586) [ 4594.789100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4594.942103] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4594.960620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4595.716162] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4597.866428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4598.686417] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4599.693395] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4600.266225] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4601.941327] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4601.944580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4602.033399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4602.035875] Lustre: Skipped 1 previous similar message [ 4603.183158] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4604.166804] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4604.634113] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 4610.986455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 4613.035766] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4613.035884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4613.036282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4613.036284] Lustre: Skipped 1 previous similar message [ 4617.972905] Lustre: server umount lustre-OST0000 complete [ 4622.987590] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4622.990140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4622.993449] Lustre: Skipped 1 previous similar message [ 4622.995036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4622.996963] Lustre: Skipped 1 previous similar message [ 4624.992208] Lustre: server umount lustre-MDT0000 complete [ 4625.951793] LustreError: 17208:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712765619 with bad export cookie 18143196286468897960 [ 4625.953823] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4625.958548] LustreError: 17208:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4626.055937] Lustre: server umount lustre-MDT0001 complete [ 4628.949658] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 4629.425540] Key type lgssc unregistered [ 4629.498390] LNet: 18002:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4629.500643] LNet: Removed LNI 192.168.202.152@tcp [ 4636.213840] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 4640.833792] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4640.981144] alg: No test for adler32 (adler32-zlib) [ 4641.732183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4641.787922] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 4641.831468] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 4641.833450] LNet: Accept secure, port 988 [ 4643.361189] Key type lgssc registered [ 4643.543101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4645.319902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4645.325812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4646.441959] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4646.452814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4647.148729] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4649.228674] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4649.232557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4649.332177] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4649.334674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4650.516933] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4651.502652] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4653.164527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4653.250826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4653.973675] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4656.251788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4661.260106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4661.260456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4661.260532] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4661.267590] Lustre: Skipped 1 previous similar message [ 4662.315765] Lustre: server umount lustre-OST0000 complete [ 4665.500566] LustreError: 23135:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712765658 with bad export cookie 6371895750273262474 [ 4665.500858] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 4665.500870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4665.501198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4665.501199] Lustre: Skipped 1 previous similar message [ 4669.314169] Lustre: server umount lustre-MDT0000 complete [ 4670.265778] LustreError: 23135:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712765663 with bad export cookie 6371895750273261998 [ 4670.266844] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4670.272436] LustreError: 23135:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4670.363052] Lustre: server umount lustre-MDT0001 complete [ 4673.195388] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 4673.649579] Key type lgssc unregistered [ 4673.709380] LNet: 24309:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4673.711359] LNet: Removed LNI 192.168.202.152@tcp [ 4681.509880] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4681.683384] alg: No test for adler32 (adler32-zlib) [ 4682.434123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4682.489061] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 4682.536151] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 4682.537478] LNet: Accept secure, port 988 [ 4684.063223] Key type lgssc registered [ 4684.259863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4686.083596] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4686.086371] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4686.090610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4703.148278] LustreError: 26181:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.202.152@tcp: failed processing log, type 1: rc = -5 [ 4739.151212] LustreError: 26181:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.202.152@tcp: failed processing log, type 4: rc = -110 [ 4889.186155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4891.757634] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4894.200535] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4898.273417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4903.389578] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4903.396169] LustreError: Skipped 1 previous similar message [ 4906.759416] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4906.785249] LustreError: Skipped 1 previous similar message [ 4906.842705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4906.899075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4909.183705] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4911.849933] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4911.883492] LustreError: Skipped 3 previous similar messages [ 4915.783142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4916.153818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4918.593500] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4921.804826] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4923.710236] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4931.121601] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4931.136715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4931.147942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4933.765005] Lustre: server umount lustre-OST0000 complete [ 4936.107882] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4936.115210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4936.131271] Lustre: Skipped 1 previous similar message [ 4936.138752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4936.144327] Lustre: Skipped 1 previous similar message [ 4937.830024] Lustre: server umount lustre-MDT0000 complete [ 4939.481861] LustreError: 26188:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712765932 with bad export cookie 3645112985708347054 [ 4939.487763] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4939.496560] LustreError: 26188:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4939.669635] Lustre: server umount lustre-MDT0001 complete [ 4944.930991] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 4945.975254] Key type lgssc unregistered [ 4946.088426] LNet: 30003:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4946.094163] LNet: Removed LNI 192.168.202.152@tcp [ 4959.559109] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4959.863435] alg: No test for adler32 (adler32-zlib) [ 4960.618183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4960.732095] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 4960.799417] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 4960.803699] LNet: Accept secure, port 988 [ 4962.365215] Key type lgssc registered [ 4962.865535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4967.563604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4967.571103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4968.863228] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4968.919079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4970.628579] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4975.715003] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4975.935564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4977.818518] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4980.652272] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4981.996435] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4986.083850] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4986.099300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4986.321656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4989.145071] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4991.548154] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4994.324633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4999.324076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4999.325397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4999.337290] Lustre: Skipped 2 previous similar messages [ 5001.352364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5001.354746] Lustre: Skipped 2 previous similar messages [ 5002.895749] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 5004.919399] Lustre: server umount lustre-MDT0000 complete [ 5005.914193] LustreError: 31890:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.52@tcp arrived at 1712765999 with bad export cookie 1915375758318780701 [ 5006.363790] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5006.367938] LustreError: Skipped 2 previous similar messages [ 5006.599596] LustreError: 31890:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712765999 with bad export cookie 1915375758318780043 [ 5006.603812] LustreError: 166-1: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5006.612821] LustreError: 31890:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5011.371617] LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5011.371783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5011.372025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5011.380839] LustreError: Skipped 1 previous similar message [ 5012.668606] Lustre: server umount lustre-OST0000 complete [ 5015.394228] LustreError: 307:0:(client.c:1281:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800a23f5500 x1795965272994432/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 5015.471577] Lustre: server umount lustre-MDT0001 complete [ 5018.539407] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 5018.998697] Key type lgssc unregistered [ 5019.057457] LNet: 3313:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5019.061231] LNet: Removed LNI 192.168.202.152@tcp [ 5027.440041] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5027.618192] alg: No test for adler32 (adler32-zlib) [ 5028.369223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5028.472651] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 5028.528056] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 5028.530967] LNet: Accept secure, port 988 [ 5030.069687] Key type lgssc registered [ 5030.342192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5032.791456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5032.800566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5033.962433] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5033.978407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5034.858117] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5037.416483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5037.569415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5038.736916] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5040.703240] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5041.776364] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5045.330835] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5045.338754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5045.505333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5047.886549] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5049.965442] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5059.499723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5062.635768] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5062.639460] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5062.645091] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5064.507883] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5064.508307] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5064.513546] Lustre: Skipped 1 previous similar message [ 5066.924714] Lustre: server umount lustre-MDT0001 complete [ 5069.516997] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5069.517368] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5069.517380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5069.517943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5069.517944] Lustre: Skipped 1 previous similar message [ 5074.524219] LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5074.526650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5074.788438] Lustre: server umount lustre-OST0000 complete [ 5076.308692] Lustre: server umount lustre-MDT0000 complete [ 5081.023297] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 5081.955512] Key type lgssc unregistered [ 5082.092783] LNet: 9058:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5082.099973] LNet: Removed LNI 192.168.202.152@tcp [ 5097.269190] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 5104.833674] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5105.164707] alg: No test for adler32 (adler32-zlib) [ 5105.924369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5106.006890] Lustre: Lustre: Build Version: 2.15.62_1_ga509b3f [ 5106.080438] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 5106.085410] LNet: Accept secure, port 988 [ 5107.748266] Key type lgssc registered [ 5108.129788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5113.127488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5118.604872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5122.682531] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5122.695227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5125.950192] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5125.957235] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5130.251260] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5135.609104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5139.923025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5139.964498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5139.977004] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5141.240184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5141.273710] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5141.361744] Lustre: lustre-MDT0000: new disk, initializing [ 5141.411999] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5141.427784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5143.389866] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5148.586933] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5148.596826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5152.851941] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5152.865762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5152.919566] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5152.935508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5153.109022] Lustre: lustre-OST0000: new disk, initializing [ 5153.111088] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5153.114671] Lustre: Skipped 1 previous similar message [ 5153.204262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5156.121906] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5160.507966] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5160.652754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5160.656430] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5160.680677] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5160.765855] Lustre: MGS: Client c6c95a19-488a-410b-8801-cc7c782dd200 (at 192.168.202.52@tcp) reconnecting [ 5161.683020] Lustre: DEBUG MARKER: conf-sanity test_70e: @@@@@@ FAIL: mount client /mnt/lustre failed