[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 509268992 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2878856K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002350] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.006677] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009123] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012038] SELinux: Initializing. [ 0.013058] *** VALIDATE selinux *** [ 0.021255] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026367] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028122] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029094] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030096] *** VALIDATE tmpfs *** [ 0.031393] *** VALIDATE proc *** [ 0.033186] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.035238] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036135] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038026] Spectre V2 : User space: Vulnerable [ 0.039006] Speculative Store Bypass: Vulnerable [ 0.041856] debug: unmapping init [mem 0xffffffffb1c59000-0xffffffffb1c60fff] [ 0.043220] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044853] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045019] ... version: 2 [ 0.046012] ... bit width: 48 [ 0.047011] ... generic registers: 4 [ 0.048014] ... value mask: 0000ffffffffffff [ 0.049016] ... max period: 00007fffffffffff [ 0.050015] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.052275] rcu: Hierarchical SRCU implementation. [ 0.054627] smp: Bringing up secondary CPUs ... [ 0.055583] x86: Booting SMP configuration: [ 0.056026] .... node #0, CPUs: #1 #2 #3 [ 0.060008] smp: Brought up 1 node, 4 CPUs [ 0.062016] smpboot: Max logical packages: 1 [ 0.063015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.167133] node 0 deferred pages initialised in 99ms [ 0.172273] devtmpfs: initialized [ 0.174338] x86/mm: Memory block size: 128MB [ 0.178114] gcov: version magic: 0x41383552 [ 0.180377] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.184176] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.187458] pinctrl core: initialized pinctrl subsystem [ 0.189238] [ 0.189773] ************************************************************* [ 0.193018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195016] ** ** [ 0.198018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.201022] ** ** [ 0.204022] ** This means that this kernel is built to expose internal ** [ 0.206017] ** IOMMU data structures, which may compromise security on ** [ 0.209020] ** your system. ** [ 0.212021] ** ** [ 0.214017] ** If you see this message and you are not debugging the ** [ 0.216021] ** kernel, report this immediately to your vendor! ** [ 0.218020] ** ** [ 0.221018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.224034] ************************************************************* [ 0.228307] NET: Registered protocol family 16 [ 0.229499] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.233064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.237068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.241098] cpuidle: using governor menu [ 0.243016] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.245530] PCI: Using configuration type 1 for base access [ 0.248126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.256404] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.259120] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.263173] cryptd: max_cpu_qlen set to 1000 [ 0.266177] ACPI: Added _OSI(Module Device) [ 0.268016] ACPI: Added _OSI(Processor Device) [ 0.270017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.272017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.278244] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.284558] ACPI: Interpreter enabled [ 0.286061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.287023] ACPI: Using IOAPIC for interrupt routing [ 0.289152] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.291478] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.303099] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.306065] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.309028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.313090] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.319582] acpiphp: Slot [2] registered [ 0.322156] acpiphp: Slot [5] registered [ 0.324204] acpiphp: Slot [6] registered [ 0.325104] acpiphp: Slot [7] registered [ 0.327177] acpiphp: Slot [8] registered [ 0.328103] acpiphp: Slot [9] registered [ 0.329152] acpiphp: Slot [10] registered [ 0.331471] acpiphp: Slot [3] registered [ 0.334167] acpiphp: Slot [4] registered [ 0.336140] acpiphp: Slot [11] registered [ 0.338148] acpiphp: Slot [12] registered [ 0.340249] acpiphp: Slot [13] registered [ 0.342125] acpiphp: Slot [14] registered [ 0.343160] acpiphp: Slot [15] registered [ 0.345110] acpiphp: Slot [16] registered [ 0.347148] acpiphp: Slot [17] registered [ 0.349154] acpiphp: Slot [18] registered [ 0.351139] acpiphp: Slot [19] registered [ 0.354182] acpiphp: Slot [20] registered [ 0.355143] acpiphp: Slot [21] registered [ 0.357164] acpiphp: Slot [22] registered [ 0.359080] acpiphp: Slot [23] registered [ 0.361122] acpiphp: Slot [24] registered [ 0.362220] acpiphp: Slot [25] registered [ 0.364188] acpiphp: Slot [26] registered [ 0.366127] acpiphp: Slot [27] registered [ 0.368103] acpiphp: Slot [28] registered [ 0.369095] acpiphp: Slot [29] registered [ 0.371119] acpiphp: Slot [30] registered [ 0.373175] acpiphp: Slot [31] registered [ 0.375070] PCI host bridge to bus 0000:00 [ 0.377024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.380026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.382025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.385023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.388024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.392028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.394230] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.399187] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.402583] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.412824] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.418526] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.422030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.424041] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.427027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.431659] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.435885] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.439046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.441750] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.447016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.461018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.468711] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.475403] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.483025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.489031] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.505023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.519000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.526015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.533015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.559015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.575246] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.586022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.597024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.624026] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.639000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.648012] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.655016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.677016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.689913] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.701022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.710023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.733023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.746271] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.755016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.760015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.782019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.792975] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.795455] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.798542] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.802595] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.805274] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.810069] iommu: Default domain type: Passthrough [ 0.812597] SCSI subsystem initialized [ 0.815159] ACPI: bus type USB registered [ 0.817120] usbcore: registered new interface driver usbfs [ 0.819080] usbcore: registered new interface driver hub [ 0.821106] usbcore: registered new device driver usb [ 0.822159] pps_core: LinuxPPS API ver. 1 registered [ 0.824011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.827069] PTP clock support registered [ 0.829159] EDAC MC: Ver: 3.0.0 [ 0.831387] PCI: Using ACPI for IRQ routing [ 0.832971] NetLabel: Initializing [ 0.833010] NetLabel: domain hash size = 128 [ 0.834010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.836092] NetLabel: unlabeled traffic allowed by default [ 0.839252] vgaarb: loaded [ 0.841249] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.843018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.851004] clocksource: Switched to clocksource kvm-clock [ 0.953568] VFS: Disk quotas dquot_6.6.0 [ 0.955605] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.959019] *** VALIDATE ramfs *** [ 0.960409] *** VALIDATE hugetlbfs *** [ 0.962414] pnp: PnP ACPI init [ 0.965243] pnp: PnP ACPI: found 6 devices [ 0.987757] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.991843] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.994582] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.998068] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.001444] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.004597] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.009069] NET: Registered protocol family 2 [ 1.011787] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.017127] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.021267] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.027347] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.030910] TCP: Hash tables configured (established 65536 bind 65536) [ 1.033879] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.037489] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.040935] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.043729] NET: Registered protocol family 1 [ 1.048195] RPC: Registered named UNIX socket transport module. [ 1.050709] RPC: Registered udp transport module. [ 1.053339] RPC: Registered tcp transport module. [ 1.055983] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.059818] NET: Registered protocol family 44 [ 1.061665] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.064043] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.066261] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.068868] PCI: CLS 0 bytes, default 64 [ 1.070608] Unpacking initramfs... [ 2.505597] debug: unmapping init [mem 0xffff9fabbcc54000-0xffff9fabbffbffff] [ 2.509887] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.511894] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.515794] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.023560] Initialise system trusted keyrings [ 3.024743] Key type blacklist registered [ 3.027365] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.036752] zbud: loaded [ 3.040251] *** VALIDATE nfs *** [ 3.041685] *** VALIDATE nfs4 *** [ 3.043942] pstore: using deflate compression [ 3.048198] Platform Keyring initialized [ 3.172117] NET: Registered protocol family 38 [ 3.173694] Key type asymmetric registered [ 3.175109] Asymmetric key parser 'x509' registered [ 3.176913] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.179528] io scheduler mq-deadline registered [ 3.181947] io scheduler kyber registered [ 3.183913] io scheduler bfq registered [ 3.186385] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.190092] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.193407] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.196923] ACPI: Power Button [PWRF] [ 3.201214] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.207571] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.221138] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.231529] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.261042] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.290790] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.322634] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.328735] Non-volatile memory driver v1.3 [ 3.330305] Linux agpgart interface v0.103 [ 3.365840] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.369206] vda: detected capacity change from 0 to 76779520 [ 3.389228] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.393532] vdb: detected capacity change from 0 to 1073741824 [ 3.409278] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.412716] vdc: detected capacity change from 0 to 2621440000 [ 3.428781] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.432222] vdd: detected capacity change from 0 to 2621440000 [ 3.446780] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.449705] vde: detected capacity change from 0 to 4294967296 [ 3.470372] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.474330] vdf: detected capacity change from 0 to 4294967296 [ 3.481542] libphy: Fixed MDIO Bus: probed [ 3.497619] usbcore: registered new interface driver usbserial_generic [ 3.500270] usbserial: USB Serial support registered for generic [ 3.503070] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.507875] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.510196] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.513303] mousedev: PS/2 mouse device common for all mice [ 3.516552] rtc_cmos 00:05: RTC can wake from S4 [ 3.551231] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.556523] rtc_cmos 00:05: registered as rtc0 [ 3.559289] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.564222] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.568371] intel_pstate: CPU model not supported [ 3.573558] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.577662] hid: raw HID events driver (C) Jiri Kosina [ 3.580695] usbcore: registered new interface driver usbhid [ 3.582499] usbhid: USB HID core driver [ 3.584359] drop_monitor: Initializing network drop monitor service [ 3.588459] Initializing XFRM netlink socket [ 3.591265] NET: Registered protocol family 10 [ 3.595425] Segment Routing with IPv6 [ 3.597202] NET: Registered protocol family 17 [ 3.599319] mpls_gso: MPLS GSO support [ 3.606162] RAS: Correctable Errors collector initialized. [ 3.608224] AVX version of gcm_enc/dec engaged. [ 3.611663] AES CTR mode by8 optimization enabled [ 3.699173] sched_clock: Marking stable (3699006881, 0)->(4605027526, -906020645) [ 3.704117] registered taskstats version 1 [ 3.706493] Loading compiled-in X.509 certificates [ 3.708355] zswap: loaded using pool lzo/zbud [ 3.733602] Key type big_key registered [ 3.747695] Key type encrypted registered [ 3.749671] ima: No TPM chip found, activating TPM-bypass! [ 3.752180] ima: Allocated hash algorithm: sha1 [ 3.754346] ima: No architecture policies found [ 3.756449] evm: Initialising EVM extended attributes: [ 3.758631] evm: security.selinux [ 3.760188] evm: security.ima [ 3.761664] evm: security.capability [ 3.763356] evm: HMAC attrs: 0x1 [ 3.767520] rtc_cmos 00:05: setting system clock to 2026-09-15 02:01:40 UTC (1789437700) [ 3.774920] debug: unmapping init [mem 0xffffffffb2c03000-0xffffffffb2dfffff] [ 3.778448] debug: unmapping init [mem 0xffffffffb1982000-0xffffffffb1c58fff] [ 3.785153] Write protecting the kernel read-only data: 28672k [ 3.789921] debug: unmapping init [mem 0xffffffffb0003000-0xffffffffb01fffff] [ 3.793089] debug: unmapping init [mem 0xffffffffb0914000-0xffffffffb09fffff] [ 3.833797] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.842475] systemd[1]: Detected virtualization kvm. [ 3.844513] systemd[1]: Detected architecture x86-64. [ 3.846872] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.872786] systemd[1]: No hostname configured. [ 3.874912] systemd[1]: Set hostname to . [ 3.877045] random: systemd: uninitialized urandom read (16 bytes read) [ 3.880177] systemd[1]: Initializing machine ID from random generator. [ 3.926552] random: ln: uninitialized urandom read (6 bytes read) [ 4.030768] random: systemd: uninitialized urandom read (16 bytes read) [ 4.033587] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.037510] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.042392] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.694251] device-mapper: uevent: version 1.0.3 [ 4.696960] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.403076] random: fast init done [ 5.419343] virtio_net virtio0 ens2: renamed from eth0 [ 5.478499] scsi host0: ata_piix [ 5.516920] scsi host1: ata_piix [ 5.520146] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.523144] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.278273] random: crng init done [ 10.289168] random: 7 urandom warning(s) missed due to ratelimiting [ 10.359824] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.773688] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.511662] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.243474] SELinux: Disabled at runtime. [ 15.385119] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.418597] systemd[1]: Detected virtualization kvm. [ 15.427457] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.763760] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.774304] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.788827] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.794421] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.800933] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.812921] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.826349] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ 16.933396] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Paths. Starting Apply Kernel Variables... Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 17.848020] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.844256] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.860052] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.123424] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.157686] EDAC sbridge: Ver: 1.1.2 [ 22.623295] Key type dns_resolver registered [ 23.216853] NFS: Registering the id_resolver key type [ 23.218516] Key type id_resolver registered [ 23.220407] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Started D-Bus System Message Bus. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg348-server login: [ 85.577938] libcfs: loading out-of-tree module taints kernel. [ 85.624691] Key type ._llcrypt registered [ 85.626101] Key type .llcrypt registered [ 85.689661] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 101.748025] hrtimer: interrupt took 1985424 ns [ 103.836962] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 105.159655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 105.181666] alg: No test for adler32 (adler32-zlib) [ 106.702662] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 107.550307] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 109.287226] Key type lgssc registered [ 111.038676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 127.241868] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 166.449622] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 178.710307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 178.772041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 180.056563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 180.084599] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 180.177578] Lustre: lustre-MDT0000: new disk, initializing [ 180.297801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 180.333377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 184.724920] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 198.485501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 198.595393] Lustre: 6504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 198.627558] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 198.631717] Lustre: Skipped 1 previous similar message [ 198.725445] Lustre: lustre-MDT0001: new disk, initializing [ 198.799735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 198.845600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 198.862525] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 203.997839] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.575631] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 218.092230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 218.410498] Lustre: lustre-OST0000: new disk, initializing [ 218.418596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 218.431292] Lustre: 8442:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 218.558807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 224.456281] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.331196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 226.343476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 226.422737] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 238.072258] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 238.219856] Lustre: lustre-OST0001: new disk, initializing [ 238.224463] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 238.230156] Lustre: 9515:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 238.290530] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 244.683586] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.357464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 248.367366] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 248.431183] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 255.941942] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 261.740370] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 268.297829] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing check_logdir /tmp/testlogs/ [ 273.795464] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing yml_node [ 278.169559] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 281.042980] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 283.542141] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 285.444497] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 14 22:06:19 EDT 2026 [ 302.253810] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 303.889202] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 319.977316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 320.002204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 320.014892] Lustre: Skipped 3 previous similar messages [ 325.108397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 325.126381] Lustre: Skipped 3 previous similar messages [ 326.347540] Lustre: server umount lustre-MDT0000 complete [ 330.701597] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789438027 with bad export cookie 13483275449829280396 [ 330.716229] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 335.327996] LustreError: 6511:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 335.348722] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 335.351985] LustreError: 6511:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 335.358846] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 335.359481] Lustre: Skipped 4 previous similar messages [ 340.449068] LustreError: 8901:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 340.449155] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 340.474427] LustreError: 8901:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 340.499668] Lustre: Skipped 2 previous similar messages [ 341.481258] LustreError: 12457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 341.502069] LustreError: 12457:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 345.055176] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 345.308313] Lustre: server umount lustre-MDT0001 complete [ 349.827585] Lustre: server umount lustre-OST0000 complete [ 354.039279] Lustre: server umount lustre-OST0001 complete [ 360.759690] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 369.648813] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 406.588932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.922229] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 406.955166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 407.016361] Lustre: lustre-MDT0000: new disk, initializing [ 407.111775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 407.140226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 411.723323] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 422.479331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.660156] Lustre: 15068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 422.716447] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 422.722780] Lustre: Skipped 1 previous similar message [ 422.809625] Lustre: lustre-MDT0001: new disk, initializing [ 422.895599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 422.931279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 422.945471] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 427.214445] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 436.232679] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 441.683437] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 447.890159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 448.256185] Lustre: lustre-OST0000: new disk, initializing [ 448.259422] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 448.268033] Lustre: 17035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 448.388869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.976084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 449.996855] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 450.142642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 455.535090] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 465.163827] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 469.608807] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 469.855479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 475.904388] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 476.256702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 480.737431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.747517] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 480.768620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 480.768630] Lustre: Skipped 2 previous similar messages [ 480.768896] Lustre: Skipped 1 previous similar message [ 480.823131] LustreError: Skipped 1 previous similar message [ 485.284352] Lustre: server umount lustre-OST0000 complete [ 490.976875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 490.978243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 490.987724] Lustre: Skipped 1 previous similar message [ 491.004281] Lustre: Skipped 2 previous similar messages [ 496.490357] Lustre: server umount lustre-MDT0000 complete [ 500.225653] LustreError: 15061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789438196 with bad export cookie 13483275449829281929 [ 500.237592] LustreError: 15061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 500.239874] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 500.575888] Lustre: server umount lustre-MDT0001 complete [ 506.978905] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 22:10:01 (1789438201) [ 527.810551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 528.814483] LustreError: 19913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.943417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 533.990645] LustreError: 19914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.369044] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 544.082105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 544.225679] LustreError: 19913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.251984] LustreError: 19913:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 548.948342] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 557.205378] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 563.375400] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 566.013284] Lustre: Failing over lustre-MDT0000 [ 566.477755] Lustre: server umount lustre-MDT0000 complete [ 570.146266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.151272] LustreError: 20929:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.170732] Lustre: Skipped 1 previous similar message [ 570.203338] LustreError: 20929:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 585.696768] Lustre: 3639:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789438266/real 1789438266] req@ffff9fac3d120380 x1876361534924928/t0(0) o400->MGC192.168.203.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789438282 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 585.736379] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 587.269992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 596.959707] LustreError: 3637:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fac20737100 x1876361534927872/t0(0) o250->MGC192.168.203.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 597.136391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 597.141880] Lustre: Skipped 1 previous similar message [ 602.550394] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 602.601911] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 602.611709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 602.643012] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 613.890717] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 620.005678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.016475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 620.028719] Lustre: Skipped 1 previous similar message [ 620.047327] Lustre: Skipped 3 previous similar messages [ 624.525970] Lustre: server umount lustre-MDT0000 complete [ 625.122284] LustreError: 19913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 625.156617] LustreError: 19913:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 629.233323] LustreError: 19895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789438325 with bad export cookie 13483275449829283343 [ 629.248865] LustreError: 19895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 629.249044] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 629.700196] Lustre: server umount lustre-MDT0001 complete [ 639.041724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.503278] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 654.490914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 660.253934] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 671.119143] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 679.358249] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 681.845913] Lustre: Failing over lustre-MDT0000 [ 682.305653] Lustre: server umount lustre-MDT0000 complete [ 685.025982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 685.052399] Lustre: Skipped 1 previous similar message [ 690.144506] LustreError: 23247:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 690.176460] LustreError: 23247:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 700.383149] Lustre: 3638:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789438381/real 1789438381] req@ffff9fab079fa680 x1876361534956672/t0(0) o400->MGC192.168.203.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789438397 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 700.432291] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 702.380454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 710.625156] LustreError: 3637:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fab05558e00 x1876361534959488/t0(0) o250->MGC192.168.203.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 710.962411] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 715.662866] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 716.270900] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 716.281718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 716.288219] Lustre: Skipped 1 previous similar message [ 716.302487] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 724.748859] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 729.579409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 729.582642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 729.604212] Lustre: Skipped 1 previous similar message [ 729.628826] Lustre: Skipped 1 previous similar message [ 735.082651] Lustre: server umount lustre-MDT0000 complete [ 739.040651] LustreError: 23230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789438435 with bad export cookie 13483275449829284281 [ 739.055891] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 739.081316] LustreError: 23230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 739.431667] Lustre: server umount lustre-MDT0001 complete [ 749.241143] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 22:14:03 (1789438443) [ 767.146833] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 777.709083] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 813.803762] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 823.704077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 823.955857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 823.990108] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 824.079951] Lustre: lustre-MDT0000: new disk, initializing [ 824.159023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 824.164470] Lustre: Skipped 3 previous similar messages [ 824.178247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 827.390278] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 837.965668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 838.164389] Lustre: 29635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 838.685947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 843.784453] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 855.074659] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 860.900768] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 867.821652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 868.147974] Lustre: 31605:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 869.581722] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 869.649659] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 869.708381] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 869.732455] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 874.818816] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 885.765124] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 891.789587] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 915.950582] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 915.968041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 915.974762] LustreError: Skipped 1 previous similar message [ 915.982966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 915.999802] Lustre: Skipped 1 previous similar message [ 916.034016] Lustre: Skipped 3 previous similar messages [ 927.202107] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 927.460178] Lustre: server umount lustre-OST0000 complete [ 935.912765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 937.877685] Lustre: server umount lustre-MDT0000 complete [ 941.542622] LustreError: 30873:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 941.565541] LustreError: 30873:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 941.970817] LustreError: 30440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789438638 with bad export cookie 13483275449829284687 [ 941.974599] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 941.980445] LustreError: 30440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 942.352277] Lustre: server umount lustre-MDT0001 complete [ 953.210887] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 956.110947] Key type lgssc unregistered [ 956.404951] LNet: 33587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.414669] LNetError: 33587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.434393] LNet: Removed LNI 192.168.203.148@tcp [ 957.290397] Key type .llcrypt unregistered [ 957.293087] Key type ._llcrypt unregistered [ 980.769593] Key type ._llcrypt registered [ 980.771687] Key type .llcrypt registered [ 980.906898] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 981.961037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 981.976560] alg: No test for adler32 (adler32-zlib) [ 983.069443] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 983.374489] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 985.039356] Key type lgssc registered [ 986.045944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 993.166409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 993.184875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 994.436667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 998.063114] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1005.032404] LustreError: 34998:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1005.059808] LustreError: 34998:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1006.768655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1007.193898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1012.684439] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1021.414508] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1028.620888] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1037.622751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1038.101915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1045.337187] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1051.468084] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1054.292782] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1071.841977] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 22:19:26 (1789438766) [ 1073.828928] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1075.582796] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1083.011588] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 22:19:37 (1789438777) [ 1094.478800] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 22:19:48 (1789438788) [ 1096.347748] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1105.398687] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1105.404680] Lustre: Skipped 2 previous similar messages [ 1107.408754] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 22:20:01 (1789438801) [ 1109.469081] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1109.472109] Lustre: Skipped 2 previous similar messages [ 1119.673817] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1119.684683] Lustre: Skipped 4 previous similar messages [ 1135.588902] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 22:20:30 (1789438830) [ 1138.956661] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1142.644847] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1151.571617] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 22:20:46 (1789438846) [ 1153.634584] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1153.645777] Lustre: Skipped 6 previous similar messages [ 1167.799156] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 22:21:02 (1789438862) [ 1184.736808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1184.743620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1184.747015] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1184.760042] Lustre: Skipped 1 previous similar message [ 1189.878795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1189.889468] Lustre: Skipped 2 previous similar messages [ 1190.709635] Lustre: server umount lustre-OST0000 complete [ 1196.512595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1196.524353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.547590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1200.101470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1200.101875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1201.720312] Lustre: server umount lustre-MDT0000 complete [ 1205.216122] LustreError: 35726:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1205.234530] LustreError: 35726:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1205.656516] LustreError: 34980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789438902 with bad export cookie 15725290121553085773 [ 1205.668954] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1205.677575] LustreError: 34980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1206.080399] Lustre: server umount lustre-MDT0001 complete [ 1220.856774] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 1224.154176] Key type lgssc unregistered [ 1224.468166] LNet: 42844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1224.483562] LNetError: 42844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1224.499476] LNet: Removed LNI 192.168.203.148@tcp [ 1225.489409] Key type .llcrypt unregistered [ 1225.495055] Key type ._llcrypt unregistered [ 1242.798794] Key type ._llcrypt registered [ 1242.801684] Key type .llcrypt registered [ 1242.903523] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 1258.469666] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 1259.539412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1259.630857] alg: No test for adler32 (adler32-zlib) [ 1260.667645] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 1260.937777] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 1262.656313] Key type lgssc registered [ 1263.696694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1297.083781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1297.110730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1298.394541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1298.447786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1298.538507] Lustre: lustre-MDT0000: new disk, initializing [ 1298.657534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1298.676739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1303.888313] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1314.886790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1315.154945] Lustre: 45992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1315.310189] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1315.317175] Lustre: Skipped 1 previous similar message [ 1315.473611] Lustre: lustre-MDT0001: new disk, initializing [ 1315.644853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1315.715887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1315.727313] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1321.272086] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1331.446976] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1337.099029] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1344.807147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1345.143431] Lustre: lustre-OST0000: new disk, initializing [ 1345.146858] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1345.154045] Lustre: 47963:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1345.270537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1352.101905] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1352.280595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1352.296904] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1352.474793] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1361.929140] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1365.860244] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1366.034084] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1370.117134] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1370.374806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1373.152299] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1373.154663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1373.172669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1373.182125] Lustre: Skipped 1 previous similar message [ 1378.273048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1378.290713] Lustre: Skipped 1 previous similar message [ 1378.772518] Lustre: server umount lustre-OST0000 complete [ 1383.403722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1383.406079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1383.431252] Lustre: Skipped 1 previous similar message [ 1383.435098] Lustre: Skipped 1 previous similar message [ 1388.512863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1388.533740] Lustre: Skipped 2 previous similar messages [ 1393.633811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1393.655430] Lustre: Skipped 1 previous similar message [ 1397.215837] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1397.546814] Lustre: server umount lustre-MDT0000 complete [ 1402.090052] LustreError: 45985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789439098 with bad export cookie 2029037386450762316 [ 1402.097978] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1402.098149] LustreError: 45985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1402.665863] Lustre: server umount lustre-MDT0001 complete [ 1414.147931] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:25:08 (1789439108) [ 1421.590145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1421.873959] LustreError: 50425:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1421.888521] LustreError: 50425:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1421.937909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1427.426398] LustreError: 50426:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1427.724550] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1431.521387] LustreError: 50425:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1436.648253] LustreError: 50426:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1438.480461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1438.937758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1445.028489] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1454.034300] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1461.341126] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1468.787954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1469.307959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1476.322075] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1484.460187] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1499.267594] LustreError: 52978:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1500.225843] Lustre: Modifying parameter general.timeout=1 in log params [ 1502.023732] Lustre: Modifying parameter general.timeout=3 in log params [ 1502.028629] Lustre: Skipped 1 previous similar message [ 1504.821624] Lustre: Modifying parameter general.timeout=6 in log params [ 1504.829194] Lustre: Skipped 2 previous similar messages [ 1509.244790] Lustre: Modifying parameter general.timeout=11 in log params [ 1509.261683] Lustre: Skipped 4 previous similar messages [ 1517.350122] Lustre: Modifying parameter general.timeout=19 in log params [ 1517.357115] Lustre: Skipped 7 previous similar messages [ 1533.528772] Lustre: Modifying parameter general.timeout=37 in log params [ 1533.536766] Lustre: Skipped 17 previous similar messages [ 1565.580386] Lustre: Modifying parameter general.timeout=73 in log params [ 1565.589209] Lustre: Skipped 35 previous similar messages [ 1631.467039] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 22:28:45 (1789439325) [ 1632.768627] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1632.781309] Lustre: Skipped 57 previous similar messages [ 1634.858886] Lustre: 59925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1637.082391] Lustre: 60025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1640.963657] Lustre: 60242:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1640.977573] Lustre: 60242:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1674.064758] Lustre: 61796:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1674.074959] Lustre: 61796:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1674.105138] LustreError: 61796:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1679.092777] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 22:29:33 (1789439373) [ 1695.202293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1695.207538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1695.239912] Lustre: Skipped 1 previous similar message [ 1695.258412] Lustre: Skipped 2 previous similar messages [ 1699.383452] Lustre: server umount lustre-MDT0000 complete [ 1700.321909] LustreError: 50426:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1700.344298] LustreError: 50426:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1703.145706] LustreError: 50451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789439399 with bad export cookie 2029037386450763198 [ 1703.159169] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1703.162095] LustreError: 50451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1703.474232] Lustre: server umount lustre-MDT0001 complete [ 1716.683815] Lustre: server umount lustre-OST0000 complete [ 1746.246223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1746.721306] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1746.936046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1752.372543] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1762.785152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1763.032264] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1763.100995] Lustre: 64326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1767.922929] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1775.888303] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1782.685937] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1790.090556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1790.382222] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1790.676789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1790.688489] Lustre: Skipped 1 previous similar message [ 1797.797964] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1806.472578] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1808.307624] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1808.319476] Lustre: Skipped 22 previous similar messages [ 1808.369695] Lustre: 66897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1816.033090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1816.036960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1816.058187] Lustre: Skipped 1 previous similar message [ 1816.067268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1821.608691] Lustre: server umount lustre-OST0000 complete [ 1824.736234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1824.748460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1824.766245] Lustre: Skipped 1 previous similar message [ 1831.023490] Lustre: server umount lustre-MDT0000 complete [ 1831.905412] LustreError: 64339:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1835.063872] LustreError: 64319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789439531 with bad export cookie 2029037386450766299 [ 1835.071376] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1835.076136] LustreError: 64319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1835.632520] Lustre: server umount lustre-MDT0001 complete [ 1848.296289] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 1851.338489] Key type lgssc unregistered [ 1851.760403] LNet: 68105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1851.774078] LNetError: 68105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1851.793478] LNet: Removed LNI 192.168.203.148@tcp [ 1852.856275] Key type .llcrypt unregistered [ 1852.860260] Key type ._llcrypt unregistered [ 1879.376621] Key type ._llcrypt registered [ 1879.381426] Key type .llcrypt registered [ 1879.544193] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 1881.058467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1881.072310] alg: No test for adler32 (adler32-zlib) [ 1882.252812] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 1882.587792] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 1884.287588] Key type lgssc registered [ 1885.628565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1893.564882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1893.580675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1895.091417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1899.777085] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1904.616709] LustreError: 69517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1904.627859] LustreError: 69517:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1907.942779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1908.543426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1913.118403] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1921.172565] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1926.586876] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1933.105372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1933.292193] LustreError: 71342:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1933.708156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1940.307405] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1947.795731] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1956.462672] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1965.536572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1965.541390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1965.559697] Lustre: Skipped 2 previous similar messages [ 1965.574719] Lustre: Skipped 2 previous similar messages [ 1970.663694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1970.670085] Lustre: Skipped 2 previous similar messages [ 1971.486537] Lustre: server umount lustre-MDT0000 complete [ 1974.879406] LustreError: 69497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789439671 with bad export cookie 4022586913229862192 [ 1974.883948] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1974.899683] LustreError: 69497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1975.272925] Lustre: server umount lustre-MDT0001 complete [ 1989.237460] Lustre: server umount lustre-OST0000 complete [ 2016.430228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2016.838429] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2016.873892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2017.161295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2021.507597] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2030.379171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2030.472561] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2030.504509] Lustre: 74386:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2031.065209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2036.090281] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2043.706184] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2049.920494] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2056.450512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2056.640782] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2057.063949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2063.791449] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2071.168442] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2072.737771] Lustre: 76965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2074.460161] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2083.297593] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2083.301981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2083.309441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2083.809683] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2085.821759] Lustre: server umount lustre-OST0000 complete [ 2092.512095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2092.522473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2092.542518] Lustre: Skipped 1 previous similar message [ 2092.552097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2092.559799] Lustre: Skipped 1 previous similar message [ 2095.759026] Lustre: server umount lustre-MDT0000 complete [ 2099.170192] LustreError: 74399:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2099.191079] LustreError: 74399:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2099.229542] LustreError: 76968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789439795 with bad export cookie 4022586913229863025 [ 2099.237923] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2099.243294] LustreError: 76968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2099.620071] Lustre: server umount lustre-MDT0001 complete [ 2109.363371] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 2112.087732] Key type lgssc unregistered [ 2112.417649] LNet: 78174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2112.429754] LNetError: 78174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2112.455117] LNet: Removed LNI 192.168.203.148@tcp [ 2113.285095] Key type .llcrypt unregistered [ 2113.288570] Key type ._llcrypt unregistered [ 2137.724195] Key type ._llcrypt registered [ 2137.727051] Key type .llcrypt registered [ 2137.878637] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 2138.895149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2139.021697] alg: No test for adler32 (adler32-zlib) [ 2140.043446] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 2140.252337] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 2141.935254] Key type lgssc registered [ 2143.227991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2150.985769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2151.003031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.683030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2157.352607] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2163.174547] LustreError: 79583:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2163.217643] LustreError: 79583:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2166.830862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2167.452718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2171.853813] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2178.271061] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2183.140468] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2189.232819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2189.786726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2196.233395] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2206.836312] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2214.514858] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2215.938824] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2217.077386] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2219.412313] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2219.416199] Lustre: Skipped 2 previous similar messages [ 2223.571167] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2223.577028] Lustre: Skipped 4 previous similar messages [ 2232.393137] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2232.398745] Lustre: Skipped 10 previous similar messages [ 2248.855974] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2248.860549] Lustre: Skipped 19 previous similar messages [ 2281.030526] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2281.034560] Lustre: Skipped 38 previous similar messages [ 2303.654170] Lustre: 87682:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2306.208408] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 22:40:00 (1789440000) [ 2307.665261] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2309.267788] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 22:40:04 (1789440004) [ 2314.724253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2314.737693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2314.757748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2315.753691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2315.753953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2315.754511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2318.888533] Lustre: server umount lustre-OST0000 complete [ 2325.984757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2325.990186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2326.004947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2326.026413] Lustre: Skipped 2 previous similar messages [ 2328.478708] LustreError: 88180:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (00000000f269df48) refcount nonzero (0) after lock cleanup; forcing cleanup. [ 2328.564448] Lustre: server umount lustre-MDT0000 complete [ 2331.107222] LustreError: 80314:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2331.132279] LustreError: 80314:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2332.314514] LustreError: 81435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789440029 with bad export cookie 6089851538463665863 [ 2332.318729] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2332.328721] LustreError: 81435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2332.682373] Lustre: server umount lustre-MDT0001 complete [ 2343.902634] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 2346.177806] Key type lgssc unregistered [ 2346.410363] LNet: 88931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2346.417591] LNetError: 88931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2346.432755] LNet: Removed LNI 192.168.203.148@tcp [ 2347.127939] Key type .llcrypt unregistered [ 2347.130462] Key type ._llcrypt unregistered [ 2350.956988] Key type ._llcrypt registered [ 2350.963068] Key type .llcrypt registered [ 2351.058338] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2355.131930] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules [ 2356.108322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2356.131875] alg: No test for adler32 (adler32-zlib) [ 2357.156395] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 2357.392051] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 2359.039371] Key type lgssc registered [ 2359.953941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2361.007579] LustreError: 90129:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2376.807428] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 2382.934070] LustreError: 90129:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2385.439274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2385.463928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2387.015342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2391.122882] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2397.162779] LustreError: 91154:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2397.175704] LustreError: 91154:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2399.796910] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 22:41:34 (1789440094) [ 2402.296895] LustreError: 91154:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2404.547699] LustreError: 91185:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2404.562770] LustreError: 91185:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2404.571232] LustreError: 91185:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2404.925540] Lustre: server umount lustre-MDT0000 complete [ 2417.279430] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 2420.013767] Key type lgssc unregistered [ 2420.248061] LNet: 92676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2420.260645] LNetError: 92676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2420.275837] LNet: Removed LNI 192.168.203.148@tcp [ 2421.102609] Key type .llcrypt unregistered [ 2421.104752] Key type ._llcrypt unregistered [ 2436.289066] Key type ._llcrypt registered [ 2436.290262] Key type .llcrypt registered [ 2436.388371] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 2437.359909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2437.456389] alg: No test for adler32 (adler32-zlib) [ 2438.468856] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 2438.720103] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 2440.407868] Key type lgssc registered [ 2441.371564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2447.119765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2447.140447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2448.818802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2453.266817] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2459.104440] LustreError: 93889:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2459.117390] LustreError: 93889:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2461.182024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2461.874464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2465.790227] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2473.294646] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2481.102864] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2488.926252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2489.973434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2495.926909] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2503.127592] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2554.535890] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 22:44:09 (1789440249) [ 2556.206811] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2558.070619] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 22:44:12 (1789440252) [ 2563.041735] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2563.050804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2563.065825] Lustre: Skipped 2 previous similar messages [ 2563.079915] Lustre: Skipped 2 previous similar messages [ 2567.773366] Lustre: server umount lustre-MDT0000 complete [ 2568.162888] LustreError: 93888:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.180645] LustreError: 93888:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2571.113622] LustreError: 93870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789440267 with bad export cookie 18245924344022591882 [ 2571.124043] LustreError: 93870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2571.127683] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2571.391217] Lustre: server umount lustre-MDT0001 complete [ 2585.175827] Lustre: server umount lustre-OST0000 complete [ 2594.585474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2595.023849] LustreError: 98185:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.131770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2598.726548] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2600.417144] LustreError: 98186:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.522328] LustreError: 98185:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2606.618916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2607.210442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2611.500873] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2618.898373] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2624.284216] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2636.648156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2636.800894] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2636.805919] LustreError: 98176:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2636.821214] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2636.829461] LustreError: 100269:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2636.834335] LustreError: 100269:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2636.841827] LustreError: 100269:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2636.912374] Lustre: server umount lustre-OST0000 complete [ 2636.915220] LustreError: 100269:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2645.048082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2645.288673] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2645.296077] LustreError: 98174:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2645.302925] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2645.318530] LustreError: 100735:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2645.332589] LustreError: 100735:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2645.339448] LustreError: 100735:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2645.432024] Lustre: server umount lustre-OST0000 complete [ 2645.436442] LustreError: 100735:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2655.122166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2655.330370] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2655.335840] Lustre: Found index 0 for lustre-OST0000, updating log [ 2655.342912] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2655.433283] Lustre: lustre-OST0000: new disk, initializing [ 2655.441732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2655.450988] Lustre: 101269:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2655.767457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2663.667303] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2668.199726] Lustre: Failing over lustre-OST0000 [ 2668.330601] Lustre: server umount lustre-OST0000 complete [ 2673.633268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2673.644268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2673.680525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2675.169511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2675.170803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2675.205396] Lustre: Skipped 1 previous similar message [ 2677.284524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2677.300858] Lustre: Skipped 1 previous similar message [ 2682.336730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2682.345780] Lustre: Skipped 1 previous similar message [ 2683.535209] Lustre: server umount lustre-MDT0000 complete [ 2687.456523] LustreError: 98916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2687.483817] LustreError: 98916:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2687.509991] LustreError: 98213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789440384 with bad export cookie 18245924344022594626 [ 2687.512847] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2687.518621] LustreError: 98213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2688.027750] Lustre: server umount lustre-MDT0001 complete [ 2697.914301] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 22:46:32 (1789440392) [ 2711.072697] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 2720.835202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2721.172770] LustreError: 103613:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2721.278090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2724.927308] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2732.567321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2737.036748] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2739.543173] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2744.921411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2745.456645] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2746.468302] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2746.500455] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2746.506530] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2746.516192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2751.387691] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2759.303158] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2759.402206] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2759.473662] Lustre: lustre-OST0001: new disk, initializing [ 2759.477522] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2759.482415] Lustre: 105930:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2759.740607] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2759.748467] Lustre: Skipped 2 previous similar messages [ 2764.445466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2764.472503] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2764.592356] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2766.184540] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2774.314353] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2780.308482] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2784.516964] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.48@tcp (stopping) [ 2784.737028] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2784.754575] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2790.292577] Lustre: server umount lustre-MDT0001 complete [ 2790.372055] LustreError: 104712:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2790.385648] LustreError: 104712:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2798.167118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2798.253646] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2798.258727] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2798.307749] Lustre: 103602:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2799.746848] LustreError: 107477:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2799.754986] LustreError: 107477:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2799.766976] LustreError: 103606:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2799.948220] LustreError: 107495:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.148@tcp: cfg command failed: rc = -17 [ 2799.965764] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2799.965764] [ 2803.947458] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2804.238976] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2815.499405] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 22:48:29 (1789440509) [ 2833.178247] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 2848.061554] Lustre: 109544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2868.282470] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2871.534451] Lustre: 110681:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2957.283761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2957.289775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2957.307809] Lustre: Skipped 2 previous similar messages [ 2957.321061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2957.329051] Lustre: Skipped 5 previous similar messages [ 2961.256957] Lustre: server umount lustre-MDT0000 complete [ 2962.912839] LustreError: 105128:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2962.923450] LustreError: 105128:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 2964.517568] LustreError: 103595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789440661 with bad export cookie 18245924344022595557 [ 2964.519030] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2964.524497] LustreError: 103595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2964.810677] Lustre: server umount lustre-MDT0001 complete [ 2979.005335] Lustre: server umount lustre-OST0000 complete [ 2992.998708] Lustre: server umount lustre-OST0001 complete [ 2997.876851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3006.622127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3014.186190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3022.854991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3039.395900] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 3048.870435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3048.916809] Lustre: lustre-MDT0000: reset Object Index mappings [ 3049.457131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3049.462475] Lustre: Skipped 1 previous similar message [ 3054.178875] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3064.635253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3064.678037] Lustre: lustre-MDT0001: reset Object Index mappings [ 3072.213722] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3076.004418] Lustre: 115782:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3083.105668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3090.114078] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3090.904839] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3095.212415] LustreError: 116136:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3095.232099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:382 to 0x280000401:418) [ 3095.238878] LustreError: 116136:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 3098.777331] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3104.760500] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3104.767576] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3105.376751] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3112.624776] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3116.124294] Lustre: 117654:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3163.837519] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 22:54:18 (1789440858) [ 3191.780736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3191.783454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3191.795429] Lustre: Skipped 5 previous similar messages [ 3191.799191] Lustre: Skipped 4 previous similar messages [ 3196.790875] Lustre: server umount lustre-MDT0000 complete [ 3199.992511] LustreError: 117673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789440896 with bad export cookie 18245924344022642653 [ 3200.000788] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3200.004333] LustreError: 117673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3206.733180] Lustre: server umount lustre-MDT0001 complete [ 3216.711474] Lustre: server umount lustre-OST0000 complete [ 3226.425725] Lustre: server umount lustre-OST0001 complete [ 3233.855613] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 3243.908165] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 3279.931242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3280.152753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3280.176405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3280.240106] Lustre: lustre-MDT0000: new disk, initializing [ 3280.352293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3280.364348] Lustre: Skipped 3 previous similar messages [ 3280.396243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3284.459753] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3290.954345] Lustre: server umount lustre-MDT0000 complete [ 3299.794565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3300.032120] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3300.043576] Lustre: 121924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3300.051962] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3305.619597] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3311.480349] Lustre: server umount lustre-MDT0000 complete [ 3322.220525] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 22:56:56 (1789441016) [ 3324.080390] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3326.085745] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 22:57:00 (1789441020) [ 3344.602656] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 3354.577242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3359.246153] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3369.072424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3369.296550] Lustre: 123972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3369.380761] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3369.390460] Lustre: Skipped 1 previous similar message [ 3369.504595] Lustre: lustre-MDT0001: new disk, initializing [ 3369.658110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3369.675994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3374.708800] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3379.644867] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3387.238067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3387.495822] Lustre: lustre-OST0000: new disk, initializing [ 3387.502174] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3387.510435] Lustre: 125552:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3389.095117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3389.106602] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3389.269217] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3394.379675] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3405.707270] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3405.982362] Lustre: lustre-OST0001: new disk, initializing [ 3405.992701] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3406.005855] Lustre: 126422:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3407.566852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3407.571097] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3407.633891] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3412.476550] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3425.556823] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3431.422589] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3823.163042] Lustre: 132852:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3823.173920] Lustre: 132852:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3823.183525] Lustre: 132852:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3823.193758] Lustre: 132852:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3823.201926] Lustre: 132852:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3823.213413] Lustre: 132852:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4119.375341] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:10:13 (1789441813) [ 4137.441935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4137.446479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4137.469361] Lustre: Skipped 3 previous similar messages [ 4137.472467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4137.478332] Lustre: Skipped 6 previous similar messages [ 4140.538472] Lustre: server umount lustre-MDT0000 complete [ 4143.080912] LustreError: 127274:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4143.126462] LustreError: 127274:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 4144.146310] LustreError: 132865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789441840 with bad export cookie 18245924344022711218 [ 4144.150831] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4144.154323] LustreError: 132865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4148.202555] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4148.211308] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4148.217175] Lustre: Skipped 4 previous similar messages [ 4148.229152] Lustre: Skipped 4 previous similar messages [ 4150.857553] Lustre: server umount lustre-MDT0001 complete [ 4169.183135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4169.261934] Lustre: server umount lustre-OST0000 complete [ 4179.836934] Lustre: server umount lustre-OST0001 complete [ 4187.151716] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 4198.319877] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 4236.177880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4236.410280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4236.434979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4236.492267] Lustre: lustre-MDT0000: new disk, initializing [ 4236.553173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4236.557871] Lustre: Skipped 5 previous similar messages [ 4236.572389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4240.974844] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4251.944980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4252.079840] Lustre: 135914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4252.128662] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4252.132967] Lustre: Skipped 1 previous similar message [ 4252.197709] Lustre: lustre-MDT0001: new disk, initializing [ 4252.315674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4252.324685] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4255.994446] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4268.099874] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4276.621535] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4283.693883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4284.261735] Lustre: lustre-OST0000: new disk, initializing [ 4284.272099] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4284.285407] Lustre: 137884:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4284.445668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4284.453546] Lustre: Skipped 1 previous similar message [ 4286.102221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4286.108458] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4286.184174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4290.068316] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4299.825060] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4309.641577] Lustre: lustre-MDD0000: changelog on [ 4312.319919] Lustre: lustre-MDD0001: changelog on [ 6342.276577] Lustre: lustre-MDD0001: changelog off [ 6344.331231] Lustre: 135923:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6345.359256] Lustre: lustre-MDD0000: changelog off [ 6354.919983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6354.953071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6354.964956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6354.979459] Lustre: Skipped 3 previous similar messages [ 6357.810208] Lustre: server umount lustre-OST0000 complete [ 6361.569299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6361.578964] LustreError: Skipped 1 previous similar message [ 6361.584482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6361.599719] Lustre: Skipped 1 previous similar message [ 6361.606717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6365.158753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6368.073459] Lustre: server umount lustre-MDT0000 complete [ 6370.272665] LustreError: 137479:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6370.299339] LustreError: 137479:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6372.099023] LustreError: 135909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789444068 with bad export cookie 18245924344023536896 [ 6372.101581] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6372.111293] LustreError: 135909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6372.542070] Lustre: server umount lustre-MDT0001 complete [ 6384.646324] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 6387.340905] Key type lgssc unregistered [ 6387.638329] LNet: 147050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6387.646801] LNetError: 147050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6387.664720] LNet: Removed LNI 192.168.203.148@tcp [ 6388.689164] Key type .llcrypt unregistered [ 6388.690769] Key type ._llcrypt unregistered [ 6408.870886] Key type ._llcrypt registered [ 6408.872726] Key type .llcrypt registered [ 6408.990968] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 6424.187093] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 6425.126948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6425.146081] alg: No test for adler32 (adler32-zlib) [ 6426.299950] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 6426.534815] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 6428.191447] Key type lgssc registered [ 6429.357477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6466.170263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6466.205121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6467.403522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6467.443470] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6467.615636] Lustre: lustre-MDT0000: new disk, initializing [ 6467.792735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6467.838929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6471.833886] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6483.584237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6483.739401] Lustre: 150247:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6483.845472] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6483.852691] Lustre: Skipped 1 previous similar message [ 6483.993565] Lustre: lustre-MDT0001: new disk, initializing [ 6484.091115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6484.158284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6484.174495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6489.079281] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6498.795170] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6505.224689] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6511.503080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6511.798683] Lustre: lustre-OST0000: new disk, initializing [ 6511.805314] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6511.820087] Lustre: 152219:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6511.991939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6518.357163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6518.370445] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6518.504506] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6518.524491] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6528.733507] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6534.501946] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_module obdecho/obdecho [ 6536.371701] LustreError: 153223:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6536.380041] LustreError: 153223:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6537.858993] LustreError: 153319:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6550.503536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6550.513388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6550.534101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6554.594263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6554.596210] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6556.018432] Lustre: server umount lustre-MDT0000 complete [ 6559.821932] LustreError: 152217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789444256 with bad export cookie 3071333756062332428 [ 6559.829556] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6559.837909] LustreError: 152217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6560.184783] Lustre: server umount lustre-MDT0001 complete [ 6574.795538] Lustre: server umount lustre-OST0000 complete [ 6595.497408] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 6605.404374] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 6642.361357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6642.728401] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6642.769326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6642.855347] Lustre: lustre-MDT0000: new disk, initializing [ 6642.982706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6642.999385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6647.842433] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6658.751468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6658.885691] Lustre: 157033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6658.932133] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6658.936892] Lustre: Skipped 1 previous similar message [ 6659.062914] Lustre: lustre-MDT0001: new disk, initializing [ 6659.186355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6659.224763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6659.243805] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6663.857294] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6673.369756] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6679.366304] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6685.640463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6685.910571] Lustre: lustre-OST0000: new disk, initializing [ 6685.916225] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6685.927404] Lustre: 159005:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6686.036602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6691.116758] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6696.012615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6696.019530] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6696.126143] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6699.949979] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6705.220634] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6705.444735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6710.129786] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6710.373534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6716.916875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6716.917890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6716.919884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6716.919895] Lustre: Skipped 2 previous similar messages [ 6716.934877] Lustre: Skipped 2 previous similar messages [ 6718.734801] Lustre: server umount lustre-OST0000 complete [ 6725.600308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6725.603975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6725.627787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6725.634163] Lustre: Skipped 1 previous similar message [ 6728.534637] Lustre: server umount lustre-MDT0000 complete [ 6732.261551] LustreError: 157040:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6732.281787] LustreError: 157040:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6732.549997] LustreError: 159004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789444429 with bad export cookie 3071333756062333387 [ 6732.552138] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6732.572882] LustreError: 159004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6732.886280] Lustre: server umount lustre-MDT0001 complete [ 6737.212527] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 23:53:51 (1789444431) [ 6738.952107] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6741.209492] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 23:53:55 (1789444435) [ 6749.536305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6750.114672] LustreError: 161419:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6750.271456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6754.241795] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6755.309912] LustreError: 161420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6759.396402] LustreError: 161419:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6763.085382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6767.891420] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6774.994425] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6780.702915] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6787.752328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6788.168339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6788.181222] Lustre: Skipped 1 previous similar message [ 6795.081470] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6804.056685] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6814.697834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6814.699757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6814.719973] Lustre: Skipped 3 previous similar messages [ 6814.720697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6814.767258] Lustre: Skipped 4 previous similar messages [ 6817.012647] Lustre: server umount lustre-MDT0000 complete [ 6819.817461] LustreError: 161419:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6819.840335] LustreError: 161419:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6821.033084] LustreError: 161399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789444517 with bad export cookie 3071333756062334269 [ 6821.042315] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6821.048298] LustreError: 161399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6824.929847] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6824.941707] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6824.949875] Lustre: Skipped 1 previous similar message [ 6827.614601] Lustre: server umount lustre-MDT0001 complete [ 6833.707820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6845.329915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6845.537320] LustreError: 165262:0:(llog.c:1655:llog_backup()) MGC192.168.203.148@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6845.552711] Lustre: 165262:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.148@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6845.735253] LustreError: 165267:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6845.758735] LustreError: 165267:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6862.056450] LustreError: 165269:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6862.075196] LustreError: 165269:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 6896.097735] LustreError: 165269:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6896.112514] LustreError: 165269:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 25 previous similar messages [ 6906.847331] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6906.856206] Lustre: 165262:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.148@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6947.827972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6952.932242] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6953.138507] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6961.693050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6962.409614] LustreError: 165268:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6962.422689] LustreError: 165268:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 42 previous similar messages [ 6970.679134] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6971.889194] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6978.831958] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6981.118719] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6988.591062] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6991.258700] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7012.833412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7012.837090] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7012.861793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7012.862796] Lustre: Skipped 2 previous similar messages [ 7012.890473] Lustre: Skipped 4 previous similar messages [ 7013.047786] Lustre: server umount lustre-MDT0000 complete [ 7017.143129] LustreError: 161400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789444713 with bad export cookie 3071333756062340723 [ 7017.148294] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7017.154383] LustreError: 161400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7017.466931] Lustre: server umount lustre-MDT0001 complete [ 7031.542249] Lustre: server umount lustre-OST0000 complete [ 7053.640857] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 7063.621493] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 7100.607057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7100.880720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7100.905917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7100.965428] Lustre: lustre-MDT0000: new disk, initializing [ 7101.031896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7101.038329] Lustre: Skipped 1 previous similar message [ 7101.050873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7106.035177] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7117.473490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7117.608960] Lustre: 170368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7117.933429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7121.744827] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7130.926738] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7138.731683] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7146.328063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7146.620443] Lustre: lustre-OST0000: new disk, initializing [ 7146.626100] Lustre: Skipped 1 previous similar message [ 7146.629399] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7146.633380] Lustre: Skipped 2 previous similar messages [ 7146.641359] Lustre: 172341:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7154.811337] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7155.779696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7155.797068] Lustre: Skipped 1 previous similar message [ 7155.801861] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7155.918632] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7165.530217] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7171.386719] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7171.608559] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7176.394190] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7176.595191] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7181.802078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7181.805353] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7181.814143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7181.814764] Lustre: Skipped 1 previous similar message [ 7181.828482] LustreError: Skipped 1 previous similar message [ 7184.630753] Lustre: server umount lustre-OST0000 complete [ 7194.584217] Lustre: server umount lustre-MDT0000 complete [ 7197.152941] LustreError: 170380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7197.175223] LustreError: 170380:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 7198.115476] LustreError: 171166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789444894 with bad export cookie 3071333756062357558 [ 7198.120376] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7198.127905] LustreError: 171166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7198.444576] Lustre: server umount lustre-MDT0001 complete [ 7206.988374] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 00:01:41 (1789444901) [ 7213.660681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7218.054979] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7225.572535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7230.333253] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7236.909797] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7242.140529] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7247.854322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7253.686375] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7260.467542] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7268.037921] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 00:02:43 (1789444963) [ 7273.957738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7273.962173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7273.962412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7273.962417] Lustre: Skipped 4 previous similar messages [ 7273.976289] Lustre: Skipped 3 previous similar messages [ 7273.989037] LustreError: Skipped 1 previous similar message [ 7278.508855] Lustre: server umount lustre-OST0000 complete [ 7289.040244] Lustre: server umount lustre-MDT0000 complete [ 7293.011580] LustreError: 174746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789444989 with bad export cookie 3071333756062358440 [ 7293.025196] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7293.031069] LustreError: 174746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7293.467905] Lustre: server umount lustre-MDT0001 complete [ 7304.653323] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 7307.596164] Key type lgssc unregistered [ 7307.931533] LNet: 178490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7307.944148] LNetError: 178490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7307.962877] LNet: Removed LNI 192.168.203.148@tcp [ 7308.765342] Key type .llcrypt unregistered [ 7308.770775] Key type ._llcrypt unregistered [ 7327.534593] Key type ._llcrypt registered [ 7327.538776] Key type .llcrypt registered [ 7327.658100] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 7328.926746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7329.005815] alg: No test for adler32 (adler32-zlib) [ 7330.072668] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 7330.283178] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 7331.903226] Key type lgssc registered [ 7333.041186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7339.632921] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7339.656096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7355.174285] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7360.288483] LustreError: 179730:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.148@tcp: failed processing log, type 4: rc = -110 [ 7365.407632] LustreError: 179730:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7365.427292] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7365.435646] LustreError: MGC192.168.203.148@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7365.447849] LustreError: 179730:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7365.460268] LustreError: 179730:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 7365.465836] LustreError: 179730:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 7365.579956] Lustre: server umount lustre-OST0000 complete [ 7365.589169] LustreError: 179730:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7374.095727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7375.601151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7380.564226] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7386.090372] LustreError: 180229:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7386.114277] LustreError: 180229:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7390.468668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7390.994947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7396.179912] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7404.096924] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7410.405732] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7416.868510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7417.089255] LustreError: 182046:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7417.102126] Lustre: 182046:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.148@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7417.224144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7423.696574] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7431.452688] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7435.745589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7435.767652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7435.770810] Lustre: Skipped 2 previous similar messages [ 7435.804221] Lustre: Skipped 2 previous similar messages [ 7440.061285] Lustre: server umount lustre-MDT0000 complete [ 7440.867053] LustreError: 180957:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7440.889616] LustreError: 180957:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7443.749977] LustreError: 180209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789445140 with bad export cookie 6466126652441583999 [ 7443.750658] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7443.762668] LustreError: 180209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7443.959861] LustreError: 180959:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009b4309a9 x1876369108660096/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7444.070993] Lustre: server umount lustre-MDT0001 complete [ 7458.246158] Lustre: server umount lustre-OST0000 complete [ 7468.741730] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 7480.070376] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 7519.146411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7519.543097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7519.571637] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7519.726046] Lustre: lustre-MDT0000: new disk, initializing [ 7519.867058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7519.900828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7524.176085] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7534.152610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7534.275258] Lustre: 185575:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7534.328966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7534.331877] Lustre: Skipped 1 previous similar message [ 7534.387427] Lustre: lustre-MDT0001: new disk, initializing [ 7534.450171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7534.489835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7534.494319] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7539.193631] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7548.017907] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7553.622916] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7560.766025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7561.077474] Lustre: lustre-OST0000: new disk, initializing [ 7561.088647] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7561.094458] Lustre: 187539:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7561.206668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7567.618409] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7571.017510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7571.053694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7571.230134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7577.879794] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7582.512931] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7582.730159] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7587.761730] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7588.053884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7591.906264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7591.910747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7591.919781] LustreError: Skipped 1 previous similar message [ 7591.921659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7591.945541] Lustre: Skipped 1 previous similar message [ 7591.964423] Lustre: Skipped 1 previous similar message [ 7596.530903] Lustre: server umount lustre-OST0000 complete [ 7601.120734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7601.133736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7601.153481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7606.275262] Lustre: server umount lustre-MDT0000 complete [ 7607.269757] LustreError: 185581:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7607.280406] LustreError: 185581:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7610.595927] LustreError: 185565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789445307 with bad export cookie 6466126652441584720 [ 7610.609925] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7610.618804] LustreError: 185565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7611.113022] Lustre: server umount lustre-MDT0001 complete [ 7620.357185] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 00:08:35 (1789445315) [ 7638.081920] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 7641.014569] Key type lgssc unregistered [ 7641.307572] LNet: 190488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7641.314586] LNetError: 190488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7641.329026] LNet: Removed LNI 192.168.203.148@tcp [ 7642.262666] Key type .llcrypt unregistered [ 7642.267551] Key type ._llcrypt unregistered [ 7660.603034] Key type ._llcrypt registered [ 7660.606705] Key type .llcrypt registered [ 7660.723050] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 7661.541385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7661.592607] alg: No test for adler32 (adler32-zlib) [ 7662.657242] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 7662.994143] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 7664.687783] Key type lgssc registered [ 7665.807443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7673.434204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7673.455402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7674.952651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7679.583661] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7685.093615] LustreError: 191700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7685.113696] LustreError: 191700:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7689.643326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7690.045679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7695.342520] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7703.555786] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7709.708408] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7717.546025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7717.739943] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7717.742565] LustreError: 193564:0:(llog.c:1696:llog_backup()) MGC192.168.203.148@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7717.748368] Lustre: 193564:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.148@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7717.893409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7724.695586] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7732.354712] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7735.266197] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7735.272154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7735.283774] Lustre: Skipped 2 previous similar messages [ 7735.302966] Lustre: Skipped 2 previous similar messages [ 7736.290801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7736.298175] Lustre: Skipped 2 previous similar messages [ 7741.410144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7741.417130] Lustre: Skipped 2 previous similar messages [ 7746.530962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7746.536630] Lustre: Skipped 2 previous similar messages [ 7749.087131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7749.232703] Lustre: server umount lustre-MDT0000 complete [ 7751.654949] LustreError: 193692:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7751.672846] LustreError: 193692:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7752.616724] LustreError: 191681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789445449 with bad export cookie 17826794489198261835 [ 7752.634243] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7752.737271] LustreError: 192430:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002af26975 x1876369456789632/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7753.151956] Lustre: server umount lustre-MDT0001 complete [ 7768.021888] Lustre: server umount lustre-OST0000 complete [ 7778.698362] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 7789.207926] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 7825.404068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7825.660761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7825.704803] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7825.762805] Lustre: lustre-MDT0000: new disk, initializing [ 7825.827473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7825.837768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7830.026471] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7840.155403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7840.297541] Lustre: 197093:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7840.338295] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7840.341599] Lustre: Skipped 1 previous similar message [ 7840.344598] LustreError: 197875:0:(llog.c:1696:llog_backup()) MGC192.168.203.148@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7840.352086] LustreError: 197875:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7840.361321] Lustre: 197875:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.148@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7840.385888] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7840.393758] Lustre: Skipped 1 previous similar message [ 7840.498985] Lustre: lustre-MDT0001: new disk, initializing [ 7840.572672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7840.607823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7840.636249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7844.900040] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7853.836410] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7859.365535] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7866.556167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7866.780534] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7866.783484] Lustre: Skipped 1 previous similar message [ 7866.786332] LustreError: 199038:0:(llog.c:1696:llog_backup()) MGC192.168.203.148@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7866.789958] LustreError: 199038:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7866.796794] Lustre: 199038:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.148@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7866.802985] Lustre: 199038:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7866.821285] Lustre: lustre-OST0000: new disk, initializing [ 7866.824221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7866.830572] Lustre: 199061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7866.907045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7873.702917] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7876.195207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7876.202516] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7876.286408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7883.259359] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7887.550144] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7887.754233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7892.264780] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7892.428903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7897.063877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7897.066049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7897.068855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7897.076181] Lustre: Skipped 1 previous similar message [ 7897.095995] LustreError: Skipped 1 previous similar message [ 7900.821392] Lustre: server umount lustre-OST0000 complete [ 7907.300329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7907.301386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7907.304206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7907.304214] Lustre: Skipped 1 previous similar message [ 7907.318059] Lustre: Skipped 1 previous similar message [ 7911.707725] Lustre: server umount lustre-MDT0000 complete [ 7912.420916] LustreError: 197104:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7912.458240] LustreError: 197104:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7916.514537] LustreError: 197098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7916.547957] LustreError: 197098:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7916.866020] LustreError: 197086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789445613 with bad export cookie 17826794489198262556 [ 7916.874113] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7916.883191] LustreError: 197086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7917.351957] Lustre: server umount lustre-MDT0001 complete [ 7927.279439] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 00:13:42 (1789445622) [ 7943.274028] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 7952.679426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7953.164410] LustreError: 202187:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7953.236561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7957.676288] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7962.597874] LustreError: 202188:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7962.636911] LustreError: 202188:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7965.771312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7970.151857] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7972.727845] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7979.330431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7987.764695] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7998.615514] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7998.878523] Lustre: lustre-OST0001: new disk, initializing [ 7998.887511] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7998.898034] Lustre: 204494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7999.052611] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7999.065272] Lustre: Skipped 2 previous similar messages [ 8004.748821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8004.770207] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8004.879675] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8005.914300] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8017.324435] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8023.518830] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8025.588051] Lustre: DEBUG MARKER: ADD OST3 [ 8030.738389] loop: module loaded [ 8030.830443] loop0: detected capacity change from 0 to 204800000 [ 8031.093708] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8031.114286] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8031.126808] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8037.203597] loop0: detected capacity change from 0 to 204800000 [ 8038.509811] Lustre: DEBUG MARKER: STOP OST3 [ 8043.135493] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8043.388543] Lustre: lustre-OST0003: new disk, initializing [ 8043.405483] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8043.411397] Lustre: 206475:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8044.992026] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8045.208260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8045.216715] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8045.290052] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8047.009630] LustreError: 206643:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8047.848977] Lustre: Failing over lustre-OST0003 [ 8048.003855] Lustre: server umount lustre-OST0003 complete [ 8050.179825] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8050.196398] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8050.196986] LustreError: 203674:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8050.238428] LustreError: 203674:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8051.000773] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8060.406763] Lustre: DEBUG MARKER: START OST3 again [ 8067.445602] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8067.806516] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8067.810981] Lustre: Skipped 1 previous similar message [ 8067.831079] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8069.289300] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8069.880933] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8069.888708] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8074.612633] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8096.738934] Lustre: Failing over lustre-OST0003 [ 8097.061545] Lustre: server umount lustre-OST0003 complete [ 8098.798731] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8098.816959] Lustre: Skipped 1 previous similar message [ 8098.824944] LustreError: 203674:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8098.840748] LustreError: 203674:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 8119.265513] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8119.269742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8119.290708] Lustre: Skipped 4 previous similar messages [ 8119.293802] Lustre: Skipped 3 previous similar messages [ 8121.588339] Lustre: server umount lustre-MDT0000 complete [ 8125.599682] LustreError: 205321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789445822 with bad export cookie 17826794489198263438 [ 8125.602383] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8125.617924] LustreError: 205321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8132.005987] Lustre: server umount lustre-MDT0001 complete [ 8136.010093] Lustre: server umount lustre-OST0000 complete [ 8139.516629] Lustre: server umount lustre-OST0001 complete [ 8145.596729] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 8155.390316] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 8196.159333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8196.410873] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8196.469374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8196.587529] Lustre: lustre-MDT0000: new disk, initializing [ 8196.683277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8196.717827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8201.721392] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8212.458632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8212.584549] Lustre: 211959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8212.646336] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8212.651867] Lustre: Skipped 1 previous similar message [ 8212.719163] Lustre: lustre-MDT0001: new disk, initializing [ 8212.837338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8212.857938] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8216.821783] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8228.659991] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8235.894561] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8244.079784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8244.638392] Lustre: 213927:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8245.834290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8245.871304] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8246.164045] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8252.764978] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8264.145559] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8269.835784] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8270.110887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8274.890622] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8275.173268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8282.091178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8282.092886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8282.095380] LustreError: Skipped 1 previous similar message [ 8282.095789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8282.095792] Lustre: Skipped 5 previous similar messages [ 8282.129082] Lustre: Skipped 3 previous similar messages [ 8291.295409] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8291.434505] Lustre: server umount lustre-OST0000 complete [ 8301.685437] Lustre: server umount lustre-MDT0000 complete [ 8302.567150] LustreError: 212878:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8302.593394] LustreError: 212878:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 8305.482861] LustreError: 211951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789446002 with bad export cookie 17826794489198265734 [ 8305.487911] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8305.506629] LustreError: 211951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8305.939491] Lustre: server umount lustre-MDT0001 complete [ 8310.308887] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 00:20:04 (1789446004) [ 8327.582754] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 8338.978891] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 8374.536853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8374.839483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8374.871961] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8374.877882] Lustre: Skipped 1 previous similar message [ 8374.972538] Lustre: lustre-MDT0000: new disk, initializing [ 8374.980909] Lustre: Skipped 1 previous similar message [ 8375.060614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8379.604260] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8391.357511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8391.507949] Lustre: 218535:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8391.734700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8396.265822] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8406.917386] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8413.356778] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8420.879663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8421.300517] Lustre: 220503:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8422.560238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8428.256195] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8438.001580] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8443.228173] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8443.494327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8448.844840] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8449.109805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8453.089373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8453.102386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8453.130018] Lustre: Skipped 2 previous similar messages [ 8453.146098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8453.156328] Lustre: Skipped 5 previous similar messages [ 8457.621575] Lustre: server umount lustre-OST0000 complete [ 8463.328706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8463.336946] LustreError: Skipped 1 previous similar message [ 8467.579317] Lustre: server umount lustre-MDT0000 complete [ 8468.962481] LustreError: 218549:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8468.986370] LustreError: 218549:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8471.464550] LustreError: 218530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789446168 with bad export cookie 17826794489198266616 [ 8471.475534] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8471.481033] LustreError: 218530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8471.755886] Lustre: server umount lustre-MDT0001 complete [ 8480.250878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8480.764458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8480.773304] Lustre: Skipped 5 previous similar messages [ 8484.980963] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8493.779807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8499.110582] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8506.880829] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8513.013712] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8520.671559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8527.872311] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8535.467194] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8546.785593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8546.806471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8546.821370] Lustre: Skipped 3 previous similar messages [ 8551.473738] Lustre: server umount lustre-OST0000 complete [ 8569.311122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8569.422230] Lustre: server umount lustre-MDT0000 complete [ 8573.554522] LustreError: 222704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789446270 with bad export cookie 17826794489198267498 [ 8573.561310] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8573.571575] LustreError: 222704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8574.277816] Lustre: server umount lustre-MDT0001 complete [ 8673.984443] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 8676.752788] Key type lgssc unregistered [ 8677.135761] LNet: 226383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8677.150445] LNetError: 226383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8677.170655] LNet: Removed LNI 192.168.203.148@tcp [ 8678.107153] Key type .llcrypt unregistered [ 8678.111381] Key type ._llcrypt unregistered [ 8706.026174] Key type ._llcrypt registered [ 8706.029351] Key type .llcrypt registered [ 8706.198385] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 8725.170817] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 8726.248366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8726.316792] alg: No test for adler32 (adler32-zlib) [ 8727.397766] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 8727.606200] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 8729.263239] Key type lgssc registered [ 8730.375640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8768.897762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8768.932328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8770.301181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8770.462920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8770.560742] Lustre: lustre-MDT0000: new disk, initializing [ 8770.693049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8770.727382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8774.547835] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8785.576680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8785.672813] Lustre: 229736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8785.721834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8785.726158] Lustre: Skipped 1 previous similar message [ 8785.816544] Lustre: lustre-MDT0001: new disk, initializing [ 8785.934590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8786.001103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8786.020352] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8790.150686] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8799.527602] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8805.443637] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8812.661662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8812.896687] Lustre: lustre-OST0000: new disk, initializing [ 8812.900360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8812.909564] Lustre: 231703:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8812.997561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8820.136346] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8823.435966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8823.460554] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8823.716487] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8829.947316] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8834.546112] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8834.751993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8839.303346] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8839.541879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8844.263240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8844.263783] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8844.266554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8844.272132] LustreError: Skipped 1 previous similar message [ 8844.297666] Lustre: Skipped 1 previous similar message [ 8847.536299] Lustre: server umount lustre-OST0000 complete [ 8852.452167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8852.461900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8852.468933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8852.471391] Lustre: Skipped 1 previous similar message [ 8854.496356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8854.497531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8859.618649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8859.627557] Lustre: Skipped 2 previous similar messages [ 8864.738294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8864.742994] Lustre: Skipped 1 previous similar message [ 8865.248202] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8865.492805] Lustre: server umount lustre-MDT0000 complete [ 8868.941498] LustreError: 229728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789446565 with bad export cookie 14345234462514565179 [ 8868.954812] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8868.960379] LustreError: 229728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8869.316786] Lustre: server umount lustre-MDT0001 complete [ 8887.697416] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 8897.477269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8897.935685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8902.829081] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8903.148163] LustreError: 234637:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8903.172451] LustreError: 234637:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8908.258601] LustreError: 234638:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8912.015990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8912.456828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8917.770954] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8920.924883] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8927.181074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8927.517479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8933.588538] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8941.465973] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8941.609481] Lustre: lustre-OST0001: new disk, initializing [ 8941.613857] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8941.618674] Lustre: 236946:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8947.359362] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8951.426563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8951.449858] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8951.588226] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8959.241265] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8966.734544] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8979.734995] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 00:31:14 (1789446674) [ 8982.508689] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8982.512826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8982.523487] Lustre: Skipped 2 previous similar messages [ 8982.531502] Lustre: Skipped 1 previous similar message [ 8996.319129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8996.540713] Lustre: server umount lustre-MDT0000 complete [ 8997.856811] LustreError: 235365:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8997.876905] LustreError: 235365:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9000.009285] LustreError: 234618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789446696 with bad export cookie 14345234462514566061 [ 9000.013684] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9000.020067] LustreError: 234618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9000.394304] Lustre: server umount lustre-MDT0001 complete [ 9014.993116] Lustre: server umount lustre-OST0000 complete [ 9018.851087] Lustre: 227956:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789446699/real 1789446699] req@ffff9fab01214380 x1876370573602944/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789446715 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9018.870920] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9018.880943] Lustre: Skipped 1 previous similar message [ 9019.368246] Lustre: server umount lustre-OST0001 complete [ 9025.836387] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 9037.946995] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 9080.722604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9080.989391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9081.038551] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9081.103638] Lustre: lustre-MDT0000: new disk, initializing [ 9081.247367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9081.257878] Lustre: Skipped 1 previous similar message [ 9081.292441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9086.744839] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 9098.588151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9098.727437] Lustre: 240890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9098.781500] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9098.793252] Lustre: Skipped 1 previous similar message [ 9098.854328] Lustre: lustre-MDT0001: new disk, initializing [ 9099.041073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9099.058379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9103.726121] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 9113.379483] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9120.046419] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9127.696546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9128.077204] Lustre: lustre-OST0000: new disk, initializing [ 9128.082980] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9128.089444] Lustre: 242862:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9128.205720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9128.214966] Lustre: Skipped 1 previous similar message [ 9130.128328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9130.156996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9130.326572] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9135.200701] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 9144.893945] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9150.565480] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9150.795440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9155.634370] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9155.906346] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9161.190444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9161.191884] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9161.194687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9161.194696] Lustre: Skipped 10 previous similar messages [ 9161.205598] Lustre: Skipped 1 previous similar message [ 9164.097979] Lustre: server umount lustre-OST0000 complete [ 9170.913513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9174.479976] Lustre: server umount lustre-MDT0000 complete [ 9176.547244] LustreError: 240902:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9176.577159] LustreError: 240902:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9178.500960] LustreError: 240884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789446875 with bad export cookie 14345234462514568189 [ 9178.504020] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9178.514624] LustreError: 240884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9178.942829] Lustre: server umount lustre-MDT0001 complete [ 9187.837246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9188.290544] LustreError: 245079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9192.556181] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 9198.564500] LustreError: 245080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9198.605453] LustreError: 245080:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9201.804862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9202.365350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9202.378518] Lustre: Skipped 1 previous similar message [ 9206.879676] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 9215.458375] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9221.611402] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9229.263921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9235.521519] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 9243.240499] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9390.562253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9390.571783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9390.588388] Lustre: Skipped 2 previous similar messages [ 9390.597918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9390.602908] Lustre: Skipped 4 previous similar messages [ 9395.400537] Lustre: server umount lustre-OST0000 complete [ 9401.824077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9401.835179] LustreError: Skipped 1 previous similar message [ 9405.407093] Lustre: server umount lustre-MDT0000 complete [ 9408.998489] LustreError: 246321:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9409.020249] LustreError: 246321:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9409.871768] LustreError: 245060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789447106 with bad export cookie 14345234462514569078 [ 9409.883432] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9409.891710] LustreError: 245060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9410.285166] Lustre: server umount lustre-MDT0001 complete [ 9514.488602] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 9517.705756] Key type lgssc unregistered [ 9517.944864] LNet: 248690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9517.960178] LNetError: 248690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9517.984803] LNet: Removed LNI 192.168.203.148@tcp [ 9519.119160] Key type .llcrypt unregistered [ 9519.126134] Key type ._llcrypt unregistered [ 9548.180144] Key type ._llcrypt registered [ 9548.182496] Key type .llcrypt registered [ 9548.352245] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 9566.896710] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 9568.235778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9568.291157] alg: No test for adler32 (adler32-zlib) [ 9569.352479] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 9569.654903] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 9571.311871] Key type lgssc registered [ 9572.579577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9619.344055] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 9629.630201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9629.683951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9630.998828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9631.041777] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9631.174485] Lustre: lustre-MDT0000: new disk, initializing [ 9631.242948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9631.256777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9636.344338] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 9647.643145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9647.802725] Lustre: 252761:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9647.885491] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9647.895076] Lustre: Skipped 1 previous similar message [ 9647.991364] Lustre: lustre-MDT0001: new disk, initializing [ 9648.113703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9648.162084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9648.176371] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9652.514224] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 9657.789873] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9664.913366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9665.244348] Lustre: lustre-OST0000: new disk, initializing [ 9665.249472] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9665.264470] Lustre: 254394:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9665.391323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9672.113414] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 9673.295992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9673.313049] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9673.444510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9682.265966] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9682.413961] Lustre: lustre-OST0001: new disk, initializing [ 9682.417144] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9682.423656] Lustre: 255263:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9682.561787] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9689.123317] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 9692.236195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9692.245895] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9692.336987] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9699.403675] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9705.120291] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9716.209186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9716.217172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9716.217950] Lustre: Skipped 2 previous similar messages [ 9716.234784] Lustre: Skipped 3 previous similar messages [ 9721.318631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9721.323070] Lustre: Skipped 3 previous similar messages [ 9722.355064] Lustre: server umount lustre-MDT0000 complete [ 9725.730659] LustreError: 252754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789447422 with bad export cookie 1617949430144102038 [ 9725.736752] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9725.739831] LustreError: 252754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9726.125976] Lustre: server umount lustre-MDT0001 complete [ 9740.180176] Lustre: server umount lustre-OST0000 complete [ 9742.815405] Lustre: 250267:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789447423/real 1789447423] req@ffff9fac3faa9c00 x1876371456468352/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789447439 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9742.841888] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9742.850931] Lustre: Skipped 1 previous similar message [ 9743.693119] Lustre: server umount lustre-OST0001 complete [ 9756.468339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9766.590411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9775.622219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9785.968717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9807.744362] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 9819.424823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9819.456753] Lustre: lustre-MDT0000: reset Object Index mappings [ 9819.887480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9824.644156] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 9825.253394] LustreError: 260275:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9825.285423] LustreError: 260275:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9829.349240] LustreError: 260274:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9834.487395] LustreError: 260275:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9835.461909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9835.493879] Lustre: lustre-MDT0001: reset Object Index mappings [ 9836.291388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9843.317166] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 9848.214956] Lustre: 261417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9855.689699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9855.995561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9861.099774] LustreError: 261773:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9863.376701] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 9865.195762] LustreError: 262035:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9865.227442] LustreError: 262035:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9865.234805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9872.634675] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9878.004919] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9879.206835] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 9886.745961] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9890.609580] Lustre: 263286:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9897.443986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9897.457191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9897.483359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9897.807675] Lustre: server umount lustre-MDT0000 complete [ 9898.472641] LustreError: 260269:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9898.500502] LustreError: 260269:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9901.483646] LustreError: 260255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789447598 with bad export cookie 1617949430144108828 [ 9901.488786] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9901.494909] LustreError: 260255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9901.844303] Lustre: server umount lustre-MDT0001 complete [ 9916.109770] Lustre: server umount lustre-OST0000 complete [ 9919.461848] Lustre: 250269:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789447600/real 1789447600] req@ffff9fac37478e00 x1876371456518144/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789447616 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9919.497275] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9919.513306] Lustre: Skipped 3 previous similar messages [ 9919.748788] Lustre: server umount lustre-OST0001 complete [ 9939.310510] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 00:47:14 (1789447634) [ 9956.050172] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 9967.165482] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [10001.587699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10001.829801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10001.876807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10001.968059] Lustre: lustre-MDT0000: new disk, initializing [10002.047460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10002.051858] Lustre: Skipped 1 previous similar message [10002.069247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10006.270362] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10017.159603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10017.291459] Lustre: 267555:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10017.410202] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10017.420879] Lustre: Skipped 1 previous similar message [10017.551321] Lustre: lustre-MDT0001: new disk, initializing [10017.702939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10017.712263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10022.449493] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10032.902370] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10040.233092] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10048.459735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10048.834540] Lustre: lustre-OST0000: new disk, initializing [10048.841736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10048.863581] Lustre: 269524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10050.214807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10050.220993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10050.372707] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10055.571712] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10067.074824] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10072.964542] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10073.171105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10079.404422] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10079.632512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10086.377246] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10086.378298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10086.400414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10086.418865] Lustre: Skipped 1 previous similar message [10091.499062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10091.513788] Lustre: Skipped 2 previous similar messages [10096.100612] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10096.589480] Lustre: server umount lustre-OST0000 complete [10101.734298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10101.751553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10101.765784] Lustre: Skipped 1 previous similar message [10107.355663] Lustre: server umount lustre-MDT0000 complete [10111.167536] LustreError: 267545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789447807 with bad export cookie 1617949430144110921 [10111.171471] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10111.180909] LustreError: 267545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10111.528936] Lustre: server umount lustre-MDT0001 complete [10130.707280] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [10141.015923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10141.317561] LustreError: 272463:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10141.367140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10141.380451] Lustre: Skipped 2 previous similar messages [10146.254403] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10155.829593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10161.281812] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10164.221587] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10171.230992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10179.269334] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10190.013558] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10190.215541] Lustre: lustre-OST0001: new disk, initializing [10190.219326] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10190.227588] Lustre: 274773:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10192.128679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10192.137882] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10192.280268] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10197.818936] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10209.266775] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10214.072757] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10219.077824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10219.106148] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10228.199338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10228.221566] Lustre: Skipped 4 previous similar messages [10228.225827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10228.227894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10228.230875] LustreError: Skipped 1 previous similar message [10228.246305] Lustre: Skipped 5 previous similar messages [10230.596962] Lustre: server umount lustre-MDT0000 complete [10233.319791] LustreError: 272458:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10233.346193] LustreError: 272458:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [10234.574576] LustreError: 272443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789447931 with bad export cookie 1617949430144111803 [10234.591532] LustreError: 272443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10234.596990] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10234.991605] Lustre: server umount lustre-MDT0001 complete [10249.313604] Lustre: server umount lustre-OST0000 complete [10263.809643] Lustre: server umount lustre-OST0001 complete [10282.184446] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [10292.952944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10293.583442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10293.589358] Lustre: Skipped 3 previous similar messages [10298.333415] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10298.848484] LustreError: 278119:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10298.864649] LustreError: 278119:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10307.739229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10313.303235] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10316.550539] Lustre: 279259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10323.462291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10330.319555] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10339.357723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10339.639690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10344.950545] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10346.183123] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10353.885993] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10357.448629] Lustre: 281135:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10362.303855] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10362.434398] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10376.106921] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 00:54:30 (1789448070) [10395.105489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10395.123467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10395.143942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10397.953597] Lustre: server umount lustre-MDT0000 complete [10402.131923] LustreError: 278099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448098 with bad export cookie 1617949430144118222 [10402.156881] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10408.729382] Lustre: server umount lustre-MDT0001 complete [10418.960160] Lustre: server umount lustre-OST0000 complete [10422.865512] Lustre: server umount lustre-OST0001 complete [10430.086451] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [10440.313588] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [10493.842325] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [10505.533726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10505.755219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10505.793922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10505.897341] Lustre: lustre-MDT0000: new disk, initializing [10506.042965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10511.358305] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10522.937690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10523.105746] Lustre: 285664:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10523.353186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10529.007770] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10541.219972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10541.412630] Lustre: 287298:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10543.284523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10548.967745] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10560.333280] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10560.588858] Lustre: 288170:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10560.719846] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10560.722493] Lustre: Skipped 6 previous similar messages [10562.075396] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10562.095560] Lustre: Skipped 1 previous similar message [10562.225713] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10567.317224] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10577.617487] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10581.767120] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10581.771564] Lustre: Skipped 1 previous similar message [10599.222225] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 00:58:13 (1789448293) [10603.497321] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10603.507794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10603.510900] Lustre: Skipped 8 previous similar messages [10603.523573] Lustre: Skipped 11 previous similar messages [10603.685808] Lustre: server umount lustre-MDT0000 complete [10607.464656] LustreError: 285657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448304 with bad export cookie 1617949430144126146 [10607.476589] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10607.490655] LustreError: 285657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10607.847855] Lustre: server umount lustre-MDT0001 complete [10622.693835] Lustre: server umount lustre-OST0000 complete [10624.613182] Lustre: 250269:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789448305/real 1789448305] req@ffff9fab099c5f80 x1876371456780288/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789448321 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10627.132671] Lustre: server umount lustre-OST0001 complete [10637.222492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10637.691803] LustreError: 290416:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10637.705686] LustreError: 290416:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [10642.982109] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10652.419900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10658.620735] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10666.318573] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10672.115747] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10678.572609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10685.367268] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10693.823152] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10714.584226] loop1: detected capacity change from 0 to 204800000 [10714.778388] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10714.799665] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10714.818491] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10719.483511] loop1: detected capacity change from 0 to 204800000 [10724.696055] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10724.765020] LustreError: 290403:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs1801: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10724.785901] LustreError: 290403:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs1801 [10724.797857] LustreError: 290403:0:(mgs_handler.c:517:mgs_target_reg()) fs1801-MDT0001: Error checking target index: rc = -13 [10724.805936] LustreError: fs1801-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10724.812128] LustreError: 294134:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10724.819676] LustreError: 294134:0:(tgt_mount.c:2128:server_put_super()) no obd fs1801-MDT0001 [10724.824099] LustreError: 294134:0:(tgt_mount.c:116:server_deregister_mount()) fs1801-MDT0001 not registered [10724.858741] Lustre: server umount fs1801-MDT0001 complete [10724.864045] LustreError: 294134:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10732.482881] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10732.619309] Lustre: Setting parameter fs1801-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs1801-MDT0001 [10732.650154] Lustre: srv-fs1801-MDT0001: No data found on store. Initialize space. [10732.652980] Lustre: Skipped 4 previous similar messages [10732.744431] Lustre: fs1801-MDT0001: new disk, initializing [10732.748644] Lustre: Skipped 3 previous similar messages [10732.841693] Lustre: srv-fs1801-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10732.846353] Lustre: 294600:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1801-MDT0001-mdtlov: can't create new llog: rc = -115 [10732.851097] LustreError: 294600:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1801-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10737.397286] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10744.285617] Lustre: server umount fs1801-MDT0001 complete [10758.108380] loop1: detected capacity change from 0 to 204800000 [10764.272787] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10764.490782] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10765.065750] Lustre: 296183:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1801-MDT0001-mdtlov: can't create new llog: rc = -115 [10765.073223] LustreError: 296183:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1801-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10769.129327] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10774.298103] Lustre: server umount fs1801-MDT0001 complete [10779.628283] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10784.743068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10784.747754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10784.753069] Lustre: Skipped 3 previous similar messages [10784.763080] Lustre: Skipped 2 previous similar messages [10790.625506] Lustre: server umount lustre-MDT0000 complete [10800.518572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10800.657734] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10804.979837] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10806.291563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10811.950155] loop1: detected capacity change from 0 to 204800000 [10817.503351] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10817.727724] Lustre: 298630:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1801-MDT0001-mdtlov: can't create new llog: rc = -115 [10817.737543] LustreError: 298630:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1801-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10823.003083] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10827.977604] Lustre: server umount fs1801-MDT0001 complete [10851.301460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10854.697617] Lustre: server umount lustre-OST0000 complete [10872.287461] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10872.483775] Lustre: server umount lustre-MDT0000 complete [10875.479188] LustreError: 290398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448572 with bad export cookie 1617949430144128827 [10875.485838] LustreError: 290398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10875.486327] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10875.794802] Lustre: server umount lustre-MDT0001 complete [10888.501925] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [10891.910663] Key type lgssc unregistered [10892.281836] LNet: 300973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10892.300265] LNetError: 300973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10892.320514] LNet: Removed LNI 192.168.203.148@tcp [10893.122144] Key type .llcrypt unregistered [10893.128300] Key type ._llcrypt unregistered [10913.753183] Key type ._llcrypt registered [10913.757595] Key type .llcrypt registered [10913.947355] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [10914.831767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10914.892442] alg: No test for adler32 (adler32-zlib) [10915.984853] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [10916.265583] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [10918.007215] Key type lgssc registered [10919.241656] Lustre: Echo OBD driver; http://www.lustre.org/ [10927.504063] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10927.538752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10929.127162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10934.222912] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10938.343384] LustreError: 302248:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10938.368138] LustreError: 302248:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10942.841781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10943.138797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10948.270246] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10956.140555] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10961.573908] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10968.896352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10969.356470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10973.413479] LustreError: 304076:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10974.893106] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10978.788452] LustreError: 304077:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10981.965220] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10983.921113] LustreError: 304076:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10988.020424] LustreError: 304076:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10988.047117] LustreError: 304076:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10996.198722] LustreError: 304077:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10996.215427] LustreError: 304077:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11004.058482] loop1: detected capacity change from 0 to 204800000 [11004.243705] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11004.262372] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11004.286715] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11009.857243] loop1: detected capacity change from 0 to 204800000 [11015.047168] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11015.208990] Lustre: fs17442-OST0000: new disk, initializing [11015.213177] Lustre: srv-fs17442-OST0000: No data found on store. Initialize space. [11015.220940] Lustre: 305981:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11015.305746] Lustre: fs17442-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11015.420262] LustreError: 304076:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11015.434557] LustreError: 304076:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [11021.811882] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [11030.705965] Lustre: server umount fs17442-OST0000 complete [11044.359477] loop1: detected capacity change from 0 to 204800000 [11051.249839] LustreError: 304327:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11051.267461] LustreError: 304327:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 20 previous similar messages [11051.318885] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11051.444958] Lustre: fs17442-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11052.531938] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11059.000490] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [11067.884618] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11068.891135] loop2: detected capacity change from 0 to 204800000 [11069.161416] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11069.173749] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11069.205170] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11075.104789] loop2: detected capacity change from 0 to 204800000 [11080.972110] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11081.110854] LustreError: fs17442-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11081.121549] LustreError: fs17442-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11081.129381] LustreError: 309025:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11081.134117] LustreError: 309025:0:(tgt_mount.c:2128:server_put_super()) no obd fs17442-OST0001 [11081.139080] LustreError: 309025:0:(tgt_mount.c:116:server_deregister_mount()) fs17442-OST0001 not registered [11081.172198] Lustre: server umount fs17442-OST0001 complete [11081.175877] LustreError: 309025:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11092.859736] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11093.015854] Lustre: fs17442-OST0001: new disk, initializing [11093.019557] Lustre: srv-fs17442-OST0001: No data found on store. Initialize space. [11093.028395] Lustre: 309575:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11093.150298] Lustre: fs17442-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11100.897187] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [11110.747611] Lustre: server umount fs17442-OST0000 complete [11117.819987] LustreError: 304327:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11117.855765] LustreError: 304327:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [11122.068744] Lustre: server umount fs17442-OST0001 complete [11137.224039] loop1: detected capacity change from 0 to 204800000 [11143.645528] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11143.952613] Lustre: fs17442-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11150.552777] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [11158.524512] loop2: detected capacity change from 0 to 204800000 [11164.880303] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11172.326410] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [11182.329709] Lustre: server umount fs17442-OST0000 complete [11193.616532] Lustre: server umount fs17442-OST0001 complete [11206.129627] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11209.184496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11209.201654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11209.242342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11211.256685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11211.268511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11211.301160] Lustre: Skipped 1 previous similar message [11214.993792] Lustre: server umount lustre-OST0000 complete [11221.473486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11221.475399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11221.488810] Lustre: Skipped 1 previous similar message [11226.594819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11226.599041] Lustre: Skipped 2 previous similar messages [11231.717802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11231.744510] Lustre: Skipped 1 previous similar message [11235.296324] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11235.518493] Lustre: server umount lustre-MDT0000 complete [11239.193145] LustreError: 302227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448935 with bad export cookie 17004174626809603840 [11239.203967] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11239.208876] LustreError: 302227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11239.727287] Lustre: server umount lustre-MDT0001 complete [11252.342205] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [11255.514525] Key type lgssc unregistered [11255.910295] LNet: 315502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11255.916746] LNetError: 315502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11255.956851] LNet: Removed LNI 192.168.203.148@tcp [11256.875765] Key type .llcrypt unregistered [11256.880612] Key type ._llcrypt unregistered [11289.726268] Key type ._llcrypt registered [11289.728982] Key type .llcrypt registered [11289.856654] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [11304.020328] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [11305.072561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11305.092941] alg: No test for adler32 (adler32-zlib) [11306.131762] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [11306.356842] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [11308.031211] Key type lgssc registered [11309.272158] Lustre: Echo OBD driver; http://www.lustre.org/ [11345.286802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11345.323717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11346.650579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11346.758817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11346.879220] Lustre: lustre-MDT0000: new disk, initializing [11346.981350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11347.014211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11351.908439] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [11362.919381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11363.048386] Lustre: 319360:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11363.101514] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11363.107489] Lustre: Skipped 1 previous similar message [11363.206372] Lustre: lustre-MDT0001: new disk, initializing [11363.329337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11363.358749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11363.373990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11368.309534] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [11377.733780] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11383.786196] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11390.649215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11391.028162] Lustre: lustre-OST0000: new disk, initializing [11391.032543] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11391.037961] Lustre: 321327:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11391.121256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11397.951592] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [11401.272511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11401.291827] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11401.409914] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11408.506770] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11410.717809] Lustre: Mounted lustre-client - version 2.17.58_39_g975873d [11411.716287] Lustre: Unmounted lustre-client [11422.186666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11422.187522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11422.192266] LustreError: Skipped 1 previous similar message [11422.210276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11422.233966] Lustre: Skipped 1 previous similar message [11426.279501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11426.294969] Lustre: Skipped 1 previous similar message [11427.573629] Lustre: server umount lustre-OST0000 complete [11431.408134] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11431.416819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11431.422089] Lustre: Skipped 2 previous similar messages [11436.517066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11436.521979] Lustre: Skipped 2 previous similar messages [11437.492659] Lustre: server umount lustre-MDT0000 complete [11441.375260] LustreError: 319351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449138 with bad export cookie 8119751448325765229 [11441.391384] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11441.419368] LustreError: 319351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11441.862743] Lustre: server umount lustre-MDT0001 complete [11455.040127] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [11458.157859] Key type lgssc unregistered [11463.072269] LNet: 1 local NIs in recovery (showing 1): 192.168.203.148@tcp [11463.651358] LNet: 323491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11463.671251] LNetError: 323491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11539.495376] LNet: Removed LNI 192.168.203.148@tcp [11540.640830] Key type .llcrypt unregistered [11540.643115] Key type ._llcrypt unregistered [11562.365467] Key type ._llcrypt registered [11562.368726] Key type .llcrypt registered [11562.474590] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [11577.934461] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [11579.263237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11579.321904] alg: No test for adler32 (adler32-zlib) [11580.456498] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [11580.796308] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [11582.439191] Key type lgssc registered [11583.479782] Lustre: Echo OBD driver; http://www.lustre.org/ [11616.150680] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 01:15:10 (1789449310) [11623.356520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11623.383722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11624.653381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11624.698523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11624.796265] Lustre: lustre-MDT0000: new disk, initializing [11624.882479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11624.905458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11630.121862] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [11640.946184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11641.083633] Lustre: 326713:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11641.163799] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11641.172693] Lustre: Skipped 1 previous similar message [11641.271703] Lustre: lustre-MDT0001: new disk, initializing [11641.394911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11641.424859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11641.445834] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11645.785839] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [11656.004269] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11662.004539] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11668.615195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11668.894398] Lustre: lustre-OST0000: new disk, initializing [11668.899661] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11668.912787] Lustre: 328680:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11669.010131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11675.827790] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [11677.288062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11677.303816] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11677.425077] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11685.744517] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11703.266063] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11703.269762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11703.273338] LustreError: Skipped 1 previous similar message [11703.274705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11703.285386] Lustre: Skipped 1 previous similar message [11703.314512] Lustre: Skipped 1 previous similar message [11708.389336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11708.402455] Lustre: Skipped 1 previous similar message [11708.538872] Lustre: server umount lustre-OST0000 complete [11712.993216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11713.006547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11713.044087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11718.627144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11718.650751] Lustre: Skipped 3 previous similar messages [11719.134877] Lustre: server umount lustre-MDT0000 complete [11723.145037] LustreError: 326704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449419 with bad export cookie 1458322255044240596 [11723.146443] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11723.154905] LustreError: 326704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11723.628533] Lustre: server umount lustre-MDT0001 complete [11734.178757] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [11736.808198] Key type lgssc unregistered [11737.116744] LNet: 330589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11737.121912] LNetError: 330589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11737.152066] LNet: Removed LNI 192.168.203.148@tcp [11738.030710] Key type .llcrypt unregistered [11738.034400] Key type ._llcrypt unregistered [11759.585779] Key type ._llcrypt registered [11759.587920] Key type .llcrypt registered [11759.707568] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [11760.730698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11760.753735] alg: No test for adler32 (adler32-zlib) [11762.026312] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [11762.285815] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [11763.945481] Key type lgssc registered [11765.147560] Lustre: Echo OBD driver; http://www.lustre.org/ [11774.032835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11774.048349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11775.450081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11780.206637] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [11785.699152] LustreError: 331862:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11785.720300] LustreError: 331862:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11788.672129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11789.111258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11793.753498] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [11802.150933] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11808.960911] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11818.801386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11819.165695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11827.389591] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [11836.501797] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11847.440895] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11849.230489] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11854.313397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11854.326387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11854.330115] Lustre: Skipped 2 previous similar messages [11854.336340] Lustre: Skipped 2 previous similar messages [11855.108930] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.48@tcp (stopping) [11859.432511] LustreError: 334295:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11859.459848] LustreError: 334295:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11859.624888] Lustre: server umount lustre-MDT0000 complete [11862.940479] LustreError: 331842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449559 with bad export cookie 4532548561843019931 [11862.942959] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11862.954864] LustreError: 331842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11864.544448] LustreError: 333356:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11864.546949] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11864.562170] LustreError: 333356:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11864.583095] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11867.554699] LustreError: 331859:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11867.555787] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11867.581685] LustreError: 331859:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11867.604442] Lustre: Skipped 1 previous similar message [11872.680286] LustreError: 334295:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11872.686277] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11872.710702] LustreError: 334295:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11872.714290] Lustre: Skipped 1 previous similar message [11877.345290] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11877.696940] Lustre: server umount lustre-MDT0001 complete [11881.360759] Lustre: server umount lustre-OST0000 complete [11889.464391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11894.104856] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [11896.830251] Lustre: 336047:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.148@tcp [11899.158845] Lustre: server umount MGS complete [11908.873703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11909.563371] LustreError: 336622:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11909.583466] LustreError: 336622:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11909.620469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11914.641960] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [11922.923850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11923.401360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11928.494353] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [11938.044322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11938.362240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11945.098138] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [11953.069059] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11955.257575] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11961.526364] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11963.219986] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11968.997775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11969.009926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11969.018753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11969.023195] Lustre: Skipped 1 previous similar message [11969.039869] Lustre: Skipped 3 previous similar messages [11970.031977] LustreError: 338058:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11970.061216] LustreError: 338058:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11970.232914] Lustre: server umount lustre-OST0000 complete [11974.625047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11974.644443] LustreError: Skipped 1 previous similar message [11974.653901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11980.484628] Lustre: server umount lustre-MDT0000 complete [11984.349318] LustreError: 336602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449681 with bad export cookie 4532548561843020771 [11984.349883] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11984.368990] LustreError: 336602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11984.697653] Lustre: server umount lustre-MDT0001 complete [11996.331869] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [11999.523681] Key type lgssc unregistered [11999.841855] LNet: 340057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11999.853188] LNetError: 340057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11999.881892] LNet: Removed LNI 192.168.203.148@tcp [12000.713841] Key type .llcrypt unregistered [12000.716077] Key type ._llcrypt unregistered [12020.342749] Key type ._llcrypt registered [12020.347146] Key type .llcrypt registered [12020.482986] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [12039.411800] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [12041.157681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12041.228642] alg: No test for adler32 (adler32-zlib) [12042.469841] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [12042.916733] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [12044.655311] Key type lgssc registered [12045.760962] Lustre: Echo OBD driver; http://www.lustre.org/ [12088.296378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12088.362517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12089.895541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12089.970312] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12090.077468] Lustre: lustre-MDT0000: new disk, initializing [12090.172673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12090.193182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12095.103666] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [12107.582270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12107.736746] Lustre: 343214:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12107.815285] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12107.825413] Lustre: Skipped 1 previous similar message [12107.943399] Lustre: lustre-MDT0001: new disk, initializing [12108.098126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12108.132562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12108.155910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12113.260269] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [12123.368622] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12131.533393] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12140.055731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12140.492481] Lustre: lustre-OST0000: new disk, initializing [12140.496782] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12140.507994] Lustre: 345183:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12140.652229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12147.340865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12147.362160] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12147.518744] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12149.245906] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [12159.574630] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12165.162839] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12165.341632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12170.882594] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12171.155035] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12178.404682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12178.417962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12178.421769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12178.426412] Lustre: Skipped 1 previous similar message [12178.452196] LustreError: Skipped 1 previous similar message [12179.876787] Lustre: server umount lustre-OST0000 complete [12183.520759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12183.523137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12183.530251] Lustre: Skipped 1 previous similar message [12183.540403] Lustre: Skipped 2 previous similar messages [12188.641766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12188.660543] Lustre: Skipped 1 previous similar message [12189.651775] Lustre: server umount lustre-MDT0000 complete [12193.428627] LustreError: 343206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449890 with bad export cookie 9257799969452178067 [12193.436852] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12193.446378] LustreError: 343206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12193.721546] Lustre: server umount lustre-MDT0001 complete [12204.031833] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 01:24:58 (1789449898) [12220.609159] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [12230.863651] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [12267.529763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12267.693335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12267.720675] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12267.808616] Lustre: lustre-MDT0000: new disk, initializing [12267.896344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12267.922536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12272.178992] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [12282.932540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12283.055396] Lustre: 349950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12283.120495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12283.124422] Lustre: Skipped 1 previous similar message [12283.201716] Lustre: lustre-MDT0001: new disk, initializing [12283.294984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12283.340803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12283.355774] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12287.794101] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [12297.501145] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12303.439292] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12311.901919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12312.158679] Lustre: lustre-OST0000: new disk, initializing [12312.162662] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12312.167936] Lustre: 351914:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12312.247363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12320.114782] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [12321.403422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12321.417932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12321.640548] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12331.061809] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12367.844648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12367.858664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12367.866921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12367.883535] Lustre: Skipped 1 previous similar message [12372.967235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12372.977336] Lustre: Skipped 1 previous similar message [12373.942663] Lustre: server umount lustre-OST0000 complete [12378.080460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12378.090640] Lustre: Skipped 1 previous similar message [12383.205675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12383.209548] Lustre: Skipped 2 previous similar messages [12383.930489] Lustre: server umount lustre-MDT0000 complete [12387.358339] LustreError: 349942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789450084 with bad export cookie 9257799969452178949 [12387.362553] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12387.371491] LustreError: 349942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12387.793822] Lustre: server umount lustre-MDT0001 complete [12397.594167] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [12400.290459] Key type lgssc unregistered [12400.645583] LNet: 354088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12400.652062] LNetError: 354088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12400.675782] LNet: Removed LNI 192.168.203.148@tcp [12401.405741] Key type .llcrypt unregistered [12401.410155] Key type ._llcrypt unregistered [12488.621471] Key type ._llcrypt registered [12488.623878] Key type .llcrypt registered [12488.748541] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [12488.970525] Key type .llcrypt unregistered [12488.977844] Key type ._llcrypt unregistered [12560.770174] Key type ._llcrypt registered [12560.782412] Key type .llcrypt registered [12560.911823] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [12561.262422] Key type .llcrypt unregistered [12561.265320] Key type ._llcrypt unregistered [12590.027562] Key type ._llcrypt registered [12590.029904] Key type .llcrypt registered [12590.130273] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [12590.347385] Key type .llcrypt unregistered [12590.353294] Key type ._llcrypt unregistered [12615.447695] Key type ._llcrypt registered [12615.453764] Key type .llcrypt registered [12615.572875] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [12615.880538] Key type .llcrypt unregistered [12615.887984] Key type ._llcrypt unregistered [12684.680560] Key type ._llcrypt registered [12684.683841] Key type .llcrypt registered [12684.795459] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [12685.102558] Key type .llcrypt unregistered [12685.104801] Key type ._llcrypt unregistered [12710.774469] Key type ._llcrypt registered [12710.775975] Key type .llcrypt registered [12710.904913] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [12711.242721] Key type .llcrypt unregistered [12711.244536] Key type ._llcrypt unregistered [12736.171942] Key type ._llcrypt registered [12736.178682] Key type .llcrypt registered [12736.314696] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [12736.575815] Key type .llcrypt unregistered [12736.579567] Key type ._llcrypt unregistered [12763.772251] Key type ._llcrypt registered [12763.774455] Key type .llcrypt registered [12763.885593] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [12764.194586] Key type .llcrypt unregistered [12764.204530] Key type ._llcrypt unregistered [12790.138860] Key type ._llcrypt registered [12790.141245] Key type .llcrypt registered [12790.226504] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [12790.488252] Key type .llcrypt unregistered [12790.491079] Key type ._llcrypt unregistered [12817.430439] Key type ._llcrypt registered [12817.432521] Key type .llcrypt registered [12817.524188] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [12817.778405] Key type .llcrypt unregistered [12817.780368] Key type ._llcrypt unregistered [12879.672597] Key type ._llcrypt registered [12879.680888] Key type .llcrypt registered [12879.779082] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [12880.072739] Key type .llcrypt unregistered [12880.076117] Key type ._llcrypt unregistered [12907.307658] Key type ._llcrypt registered [12907.310431] Key type .llcrypt registered [12907.404440] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [12907.755389] Key type .llcrypt unregistered [12907.760924] Key type ._llcrypt unregistered [12933.148933] Key type ._llcrypt registered [12933.151477] Key type .llcrypt registered [12933.279910] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [12933.601475] Key type .llcrypt unregistered [12933.605279] Key type ._llcrypt unregistered [12988.635857] Key type ._llcrypt registered [12988.640401] Key type .llcrypt registered [12988.786859] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [13004.302459] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [13005.055044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13005.071845] alg: No test for adler32 (adler32-zlib) [13006.113079] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [13006.309586] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [13007.967227] Key type lgssc registered [13009.165935] Lustre: Echo OBD driver; http://www.lustre.org/ [13039.817151] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13039.859163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13041.265651] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13041.304281] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13041.417312] Lustre: tst250fs-MDT0000: new disk, initializing [13041.483958] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13041.505823] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13046.077586] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [13057.015522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13057.147932] Lustre: 374219:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13057.201147] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13057.205787] Lustre: Skipped 1 previous similar message [13057.295270] Lustre: tst250fs-MDT0001: new disk, initializing [13057.395574] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13057.433735] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13057.444525] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13060.827909] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [13065.575525] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13071.242038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13071.506943] Lustre: tst250fs-OST0000: new disk, initializing [13071.512426] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13071.520490] Lustre: 375853:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13071.617797] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13078.298896] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [13079.666775] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13079.676907] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13079.828889] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13089.376404] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13089.564919] Lustre: tst250fs-OST0001: new disk, initializing [13089.576173] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13089.587209] Lustre: 376724:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13089.694306] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13096.786582] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [13099.091624] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13099.100598] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13099.198235] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13102.683657] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13105.282890] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13105.291146] Lustre: Skipped 2 previous similar messages [13124.064127] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13124.074384] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13124.088304] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13125.092099] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13125.092874] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13125.132102] Lustre: Skipped 3 previous similar messages [13130.209519] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13130.215059] Lustre: Skipped 3 previous similar messages [13135.333157] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13135.341916] Lustre: Skipped 2 previous similar messages [13136.352860] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13136.607817] Lustre: server umount tst250fs-MDT0000 complete [13140.004188] LustreError: 374211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789450836 with bad export cookie 13000964899305149214 [13140.008267] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13140.018211] LustreError: 374211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13140.312339] Lustre: server umount tst250fs-MDT0001 complete [13154.043691] Lustre: server umount tst250fs-OST0000 complete [13156.319693] Lustre: 372445:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789450837/real 1789450837] req@ffff9fac3d4c3100 x1876375060416640/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789450853 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13156.365053] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13156.382755] Lustre: Skipped 2 previous similar messages [13157.754749] Lustre: server umount tst250fs-OST0001 complete [13175.926944] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [13184.365279] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [13216.612691] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 01:41:51 (1789450911) [13218.570103] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13234.752735] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [13244.646341] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [13273.823306] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12987 sec ========= 01:42:48 (1789450968) [13275.340169] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:42:50 (1789450970) === [13279.144552] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:42:53 (1789450973) === [13318.149888] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [13320.625481] Key type lgssc unregistered [13320.924118] LNet: 385851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13320.940984] LNetError: 385851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13320.956373] LNet: Removed LNI 192.168.203.148@tcp [13321.977190] Key type .llcrypt unregistered [13321.984281] Key type ._llcrypt unregistered