[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 478728724 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003175] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007893] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010133] LSM: Security Framework initializing [ 0.011065] Yama: becoming mindful. [ 0.012047] SELinux: Initializing. [ 0.013082] *** VALIDATE selinux *** [ 0.021898] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026339] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029102] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030121] *** VALIDATE tmpfs *** [ 0.031506] *** VALIDATE proc *** [ 0.033056] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035294] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036188] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.042501] debug: unmapping init [mem 0xffffffffb3459000-0xffffffffb3460fff] [ 0.044233] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045830] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046030] ... version: 2 [ 0.047016] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049017] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051014] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.053301] rcu: Hierarchical SRCU implementation. [ 0.055537] smp: Bringing up secondary CPUs ... [ 0.056640] x86: Booting SMP configuration: [ 0.057022] .... node #0, CPUs: #1 #2 #3 [ 0.063266] smp: Brought up 1 node, 4 CPUs [ 0.065016] smpboot: Max logical packages: 1 [ 0.066012] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.146276] node 0 deferred pages initialised in 78ms [ 0.150011] devtmpfs: initialized [ 0.151258] x86/mm: Memory block size: 128MB [ 0.153876] gcov: version magic: 0x41383552 [ 0.155277] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157276] pinctrl core: initialized pinctrl subsystem [ 0.158265] [ 0.158842] ************************************************************* [ 0.159018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160018] ** ** [ 0.161019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162016] ** ** [ 0.163016] ** This means that this kernel is built to expose internal ** [ 0.164013] ** IOMMU data structures, which may compromise security on ** [ 0.165018] ** your system. ** [ 0.166024] ** ** [ 0.167015] ** If you see this message and you are not debugging the ** [ 0.168014] ** kernel, report this immediately to your vendor! ** [ 0.169012] ** ** [ 0.170011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171016] ************************************************************* [ 0.172720] NET: Registered protocol family 16 [ 0.173465] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.174069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.175119] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.177034] cpuidle: using governor menu [ 0.180416] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.183791] PCI: Using configuration type 1 for base access [ 0.185152] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.195149] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196047] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.198081] cryptd: max_cpu_qlen set to 1000 [ 0.200500] ACPI: Added _OSI(Module Device) [ 0.201014] ACPI: Added _OSI(Processor Device) [ 0.202013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.203014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.207314] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.214607] ACPI: Interpreter enabled [ 0.215081] ACPI: PM: (supports S0 S3 S4 S5) [ 0.216020] ACPI: Using IOAPIC for interrupt routing [ 0.218178] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.222519] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.233588] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.237073] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.240030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.245138] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.251812] acpiphp: Slot [2] registered [ 0.253149] acpiphp: Slot [5] registered [ 0.255270] acpiphp: Slot [6] registered [ 0.257257] acpiphp: Slot [7] registered [ 0.258219] acpiphp: Slot [8] registered [ 0.260155] acpiphp: Slot [9] registered [ 0.261220] acpiphp: Slot [10] registered [ 0.263252] acpiphp: Slot [3] registered [ 0.265185] acpiphp: Slot [4] registered [ 0.267134] acpiphp: Slot [11] registered [ 0.268163] acpiphp: Slot [12] registered [ 0.270141] acpiphp: Slot [13] registered [ 0.272131] acpiphp: Slot [14] registered [ 0.273137] acpiphp: Slot [15] registered [ 0.275192] acpiphp: Slot [16] registered [ 0.276142] acpiphp: Slot [17] registered [ 0.278118] acpiphp: Slot [18] registered [ 0.279192] acpiphp: Slot [19] registered [ 0.280109] acpiphp: Slot [20] registered [ 0.282105] acpiphp: Slot [21] registered [ 0.284112] acpiphp: Slot [22] registered [ 0.285126] acpiphp: Slot [23] registered [ 0.287090] acpiphp: Slot [24] registered [ 0.288081] acpiphp: Slot [25] registered [ 0.290137] acpiphp: Slot [26] registered [ 0.291114] acpiphp: Slot [27] registered [ 0.293118] acpiphp: Slot [28] registered [ 0.295122] acpiphp: Slot [29] registered [ 0.297116] acpiphp: Slot [30] registered [ 0.298096] acpiphp: Slot [31] registered [ 0.300079] PCI host bridge to bus 0000:00 [ 0.301017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.304026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.307024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.309080] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.312026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.315026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.317180] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.320108] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.324235] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.335024] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.340116] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.343034] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.345024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.348047] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.351712] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.355838] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.358054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.361350] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.367017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.380998] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.386013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.391713] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.399018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.404018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.418017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.433527] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.442024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.449021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.466019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.479490] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.487022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.494035] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.517021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.530067] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.537020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.543025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.560025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.573420] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.580021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.588021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.607026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.616522] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.622019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.629021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.642030] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.654310] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.659482] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.661480] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.664509] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.667307] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.672124] iommu: Default domain type: Passthrough [ 0.675545] SCSI subsystem initialized [ 0.678191] ACPI: bus type USB registered [ 0.681148] usbcore: registered new interface driver usbfs [ 0.684130] usbcore: registered new interface driver hub [ 0.686141] usbcore: registered new device driver usb [ 0.689209] pps_core: LinuxPPS API ver. 1 registered [ 0.691016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.695068] PTP clock support registered [ 0.697119] EDAC MC: Ver: 3.0.0 [ 0.699195] PCI: Using ACPI for IRQ routing [ 0.702030] NetLabel: Initializing [ 0.704017] NetLabel: domain hash size = 128 [ 0.706017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.708156] NetLabel: unlabeled traffic allowed by default [ 0.711119] vgaarb: loaded [ 0.712346] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.715014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.720338] clocksource: Switched to clocksource kvm-clock [ 0.826353] VFS: Disk quotas dquot_6.6.0 [ 0.828330] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.831368] *** VALIDATE ramfs *** [ 0.832715] *** VALIDATE hugetlbfs *** [ 0.834802] pnp: PnP ACPI init [ 0.836728] pnp: PnP ACPI: found 6 devices [ 0.855665] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.860148] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.862907] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.866021] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.868798] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.872211] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.875280] NET: Registered protocol family 2 [ 0.878090] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.884062] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.888311] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.893978] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.898844] TCP: Hash tables configured (established 65536 bind 65536) [ 0.902867] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.907316] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.911604] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.914523] NET: Registered protocol family 1 [ 0.917372] RPC: Registered named UNIX socket transport module. [ 0.920435] RPC: Registered udp transport module. [ 0.922126] RPC: Registered tcp transport module. [ 0.924038] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.926855] NET: Registered protocol family 44 [ 0.928887] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.931357] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.933641] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.936239] PCI: CLS 0 bytes, default 64 [ 0.938944] Unpacking initramfs... [ 2.350449] debug: unmapping init [mem 0xffff8f2bfcc54000-0xffff8f2bfffbffff] [ 2.354766] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.357124] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.359975] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.859560] Initialise system trusted keyrings [ 2.861413] Key type blacklist registered [ 2.863802] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.874290] zbud: loaded [ 2.878594] *** VALIDATE nfs *** [ 2.880238] *** VALIDATE nfs4 *** [ 2.882314] pstore: using deflate compression [ 2.886280] Platform Keyring initialized [ 2.989343] NET: Registered protocol family 38 [ 2.991265] Key type asymmetric registered [ 2.992848] Asymmetric key parser 'x509' registered [ 2.994682] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.998298] io scheduler mq-deadline registered [ 3.000298] io scheduler kyber registered [ 3.002271] io scheduler bfq registered [ 3.004781] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.007629] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.010834] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.014482] ACPI: Power Button [PWRF] [ 3.019994] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.026694] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.037943] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.047905] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.060785] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.091347] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.123013] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.128610] Non-volatile memory driver v1.3 [ 3.130180] Linux agpgart interface v0.103 [ 3.166267] virtio_blk virtio1: [vda] 145912 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.169513] vda: detected capacity change from 0 to 74706944 [ 3.185645] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.189234] vdb: detected capacity change from 0 to 1073741824 [ 3.203220] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.206930] vdc: detected capacity change from 0 to 2621440000 [ 3.222680] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.225431] vdd: detected capacity change from 0 to 2621440000 [ 3.243883] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.247089] vde: detected capacity change from 0 to 4294967296 [ 3.262429] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.265672] vdf: detected capacity change from 0 to 4294967296 [ 3.274513] libphy: Fixed MDIO Bus: probed [ 3.279907] usbcore: registered new interface driver usbserial_generic [ 3.282604] usbserial: USB Serial support registered for generic [ 3.284319] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.287935] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.289825] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.292151] mousedev: PS/2 mouse device common for all mice [ 3.295367] rtc_cmos 00:05: RTC can wake from S4 [ 3.298566] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.299784] rtc_cmos 00:05: registered as rtc0 [ 3.306126] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.307531] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.310423] intel_pstate: CPU model not supported [ 3.317183] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.321962] hid: raw HID events driver (C) Jiri Kosina [ 3.324507] usbcore: registered new interface driver usbhid [ 3.326749] usbhid: USB HID core driver [ 3.328985] drop_monitor: Initializing network drop monitor service [ 3.333275] Initializing XFRM netlink socket [ 3.336150] NET: Registered protocol family 10 [ 3.340291] Segment Routing with IPv6 [ 3.342257] NET: Registered protocol family 17 [ 3.344985] mpls_gso: MPLS GSO support [ 3.351582] RAS: Correctable Errors collector initialized. [ 3.354492] AVX version of gcm_enc/dec engaged. [ 3.356525] AES CTR mode by8 optimization enabled [ 3.432695] sched_clock: Marking stable (3432673625, 0)->(4401530281, -968856656) [ 3.437172] registered taskstats version 1 [ 3.440311] Loading compiled-in X.509 certificates [ 3.442516] zswap: loaded using pool lzo/zbud [ 3.465091] Key type big_key registered [ 3.480903] Key type encrypted registered [ 3.482612] ima: No TPM chip found, activating TPM-bypass! [ 3.484555] ima: Allocated hash algorithm: sha1 [ 3.486797] ima: No architecture policies found [ 3.488471] evm: Initialising EVM extended attributes: [ 3.490084] evm: security.selinux [ 3.491417] evm: security.ima [ 3.492306] evm: security.capability [ 3.493835] evm: HMAC attrs: 0x1 [ 3.496297] rtc_cmos 00:05: setting system clock to 2026-08-18 17:58:28 UTC (1787075908) [ 3.504556] debug: unmapping init [mem 0xffffffffb4403000-0xffffffffb45fffff] [ 3.508913] debug: unmapping init [mem 0xffffffffb3182000-0xffffffffb3458fff] [ 3.518122] Write protecting the kernel read-only data: 28672k [ 3.521910] debug: unmapping init [mem 0xffffffffb1803000-0xffffffffb19fffff] [ 3.525038] debug: unmapping init [mem 0xffffffffb2114000-0xffffffffb21fffff] [ 3.560277] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.570125] systemd[1]: Detected virtualization kvm. [ 3.572375] systemd[1]: Detected architecture x86-64. [ 3.574675] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.604440] systemd[1]: No hostname configured. [ 3.606419] systemd[1]: Set hostname to . [ 3.608844] random: systemd: uninitialized urandom read (16 bytes read) [ 3.611750] systemd[1]: Initializing machine ID from random generator. [ 3.667253] random: ln: uninitialized urandom read (6 bytes read) [ 3.758996] random: systemd: uninitialized urandom read (16 bytes read) [ 3.762735] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.769196] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.773606] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.481942] device-mapper: uevent: version 1.0.3 [ 4.484623] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ 5.181655] random: fast init done [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.240266] virtio_net virtio0 ens2: renamed from eth0 [ 5.271186] scsi host0: ata_piix [ 5.341140] scsi host1: ata_piix [ 5.342853] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.346501] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.693930] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.998335] random: crng init done [ 10.000393] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.479995] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.683202] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.981957] SELinux: Disabled at runtime. [ 12.044939] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.057582] systemd[1]: Detected virtualization kvm. [ 12.059510] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.564204] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.567442] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.572619] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.576616] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.579183] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.587417] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.595123] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ 12.701830] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.022532] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.320067] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.334746] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.474355] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.487762] EDAC sbridge: Ver: 1.1.2 [ 15.023874] Key type dns_resolver registered [ 15.325164] NFS: Registering the id_resolver key type [ 15.327385] Key type id_resolver registered [ 15.329264] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg449-server login: [ 46.165131] libcfs: loading out-of-tree module taints kernel. [ 46.253273] Key type ._llcrypt registered [ 46.256870] Key type .llcrypt registered [ 46.357697] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 62.589022] hrtimer: interrupt took 2670123 ns [ 67.138105] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 69.181800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 69.222346] alg: No test for adler32 (adler32-zlib) [ 70.991687] Lustre: Lustre: Build Version: 2.17.57_1_g711d631 [ 72.172896] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 73.936930] Key type lgssc registered [ 75.830370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 93.869239] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 133.437489] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 145.480603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 145.539175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 146.729368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 146.776929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 146.875085] Lustre: lustre-MDT0000: new disk, initializing [ 146.941982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 146.958988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 151.587514] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 166.095490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 166.206477] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 166.262318] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 166.270730] Lustre: Skipped 1 previous similar message [ 166.416223] Lustre: lustre-MDT0001: new disk, initializing [ 166.563070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 166.592546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 166.602597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 171.855907] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.030414] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 187.292419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 187.578419] Lustre: lustre-OST0000: new disk, initializing [ 187.587106] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 187.599960] Lustre: 8457:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 187.691330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.580920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 193.591963] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 193.664266] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 194.004965] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.312261] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 208.467416] Lustre: lustre-OST0001: new disk, initializing [ 208.471050] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 208.481794] Lustre: 9530:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 208.538533] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.874621] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.175405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 218.182508] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 218.260082] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 225.058833] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 235.061042] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 241.860147] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing check_logdir /tmp/testlogs/ [ 246.980265] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing yml_node [ 251.360660] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 253.757313] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 255.989645] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 257.689537] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Aug 18 14:02:41 EDT 2026 [ 272.775709] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 273.976376] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 289.765071] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 289.771179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 289.775901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 289.791814] Lustre: Skipped 3 previous similar messages [ 289.798947] Lustre: Skipped 2 previous similar messages [ 294.884267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 294.900844] Lustre: Skipped 3 previous similar messages [ 295.209413] Lustre: server umount lustre-MDT0000 complete [ 298.945802] LustreError: 10345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787076203 with bad export cookie 17111150853244588884 [ 298.948060] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 298.971134] LustreError: 10345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 300.002273] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.011106] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 300.041944] Lustre: Skipped 1 previous similar message [ 300.061323] Lustre: Skipped 1 previous similar message [ 303.016972] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 303.027024] Lustre: Skipped 1 previous similar message [ 308.129122] LustreError: 9538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 308.133365] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 308.161826] LustreError: 9538:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 308.179141] Lustre: Skipped 1 previous similar message [ 313.312102] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 313.316149] LustreError: 6515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 313.346069] LustreError: 6515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 313.764878] Lustre: server umount lustre-MDT0001 complete [ 317.927560] Lustre: server umount lustre-OST0000 complete [ 322.256541] Lustre: server umount lustre-OST0001 complete [ 329.416693] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 339.251276] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 376.259761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 376.664033] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 376.718742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 376.837222] Lustre: lustre-MDT0000: new disk, initializing [ 376.907717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 376.921369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 382.345990] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 394.470500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.625753] Lustre: 15090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 394.695354] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 394.702230] Lustre: Skipped 1 previous similar message [ 394.895582] Lustre: lustre-MDT0001: new disk, initializing [ 395.122201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 395.181241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 395.194521] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 400.103384] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 410.118495] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 416.807143] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 423.627340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 424.082166] Lustre: lustre-OST0000: new disk, initializing [ 424.088389] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 424.097916] Lustre: 17061:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 424.208843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 425.531336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 425.543943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 425.670519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 430.269360] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 441.054117] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 447.169475] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 447.433544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 452.958978] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 453.171615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 456.673699] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 456.685485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 456.691390] LustreError: Skipped 1 previous similar message [ 456.697077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 456.710048] Lustre: Skipped 1 previous similar message [ 456.735804] Lustre: Skipped 3 previous similar messages [ 461.246945] Lustre: server umount lustre-OST0000 complete [ 466.915186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.915568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 466.938055] Lustre: Skipped 1 previous similar message [ 472.035430] LustreError: 15096:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 472.072577] LustreError: 15096:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 473.495671] Lustre: server umount lustre-MDT0000 complete [ 476.130260] LustreError: 15100:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 476.149246] LustreError: 15100:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 478.185838] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787076383 with bad export cookie 17111150853244590417 [ 478.188212] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 478.198932] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 478.647666] Lustre: server umount lustre-MDT0001 complete [ 485.714767] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 14:06:28 (1787076388) [ 507.430437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 507.731528] LustreError: 19936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.809297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 513.453413] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 517.119120] LustreError: 19937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.160088] LustreError: 19937:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 524.139049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 529.503257] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 537.831663] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 544.700566] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 547.433471] Lustre: Failing over lustre-MDT0000 [ 549.749483] Lustre: server umount lustre-MDT0000 complete [ 550.373729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 550.379521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 550.385329] Lustre: Skipped 1 previous similar message [ 550.390247] LustreError: 19931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.411403] LustreError: 19931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 566.752748] Lustre: 3641:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787076455/real 1787076455] req@ffff8f2b47489c00 x1873884979817856/t0(0) o400->MGC192.168.204.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787076471 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 566.777324] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 568.664541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 576.993640] LustreError: 3639:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8f2b4579e680 x1873884979821056/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 577.225039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 577.232224] Lustre: Skipped 1 previous similar message [ 582.432699] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 582.634774] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 582.642924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 582.681322] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 593.303624] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 597.986736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 597.999542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 598.022673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 598.031561] Lustre: Skipped 2 previous similar messages [ 603.983751] Lustre: server umount lustre-MDT0000 complete [ 607.206819] LustreError: 21271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 607.246306] LustreError: 21271:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 608.029490] LustreError: 19916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787076513 with bad export cookie 17111150853244591831 [ 608.034108] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 608.052685] LustreError: 19916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 608.912610] Lustre: server umount lustre-MDT0001 complete [ 617.918747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.346836] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 632.617449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 637.791602] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 645.323930] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 651.657311] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 653.940487] Lustre: Failing over lustre-MDT0000 [ 654.259766] Lustre: server umount lustre-MDT0000 complete [ 657.060174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 657.078691] Lustre: Skipped 1 previous similar message [ 672.739031] LustreError: 24285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 672.761552] LustreError: 24285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 673.450651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 673.606729] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 673.957952] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 677.997495] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 678.007655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 678.017354] Lustre: Skipped 1 previous similar message [ 678.045537] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 678.086578] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 688.712980] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 693.729541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 693.757525] Lustre: Skipped 1 previous similar message [ 693.765938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 693.777732] Lustre: Skipped 3 previous similar messages [ 699.075559] Lustre: server umount lustre-MDT0000 complete [ 703.164326] LustreError: 23255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787076608 with bad export cookie 17111150853244592769 [ 703.173055] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 703.177922] LustreError: 23255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 703.579351] Lustre: server umount lustre-MDT0001 complete [ 713.161844] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 14:10:16 (1787076616) [ 731.004492] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 742.297390] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 786.737821] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 796.659723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 796.925090] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 796.966504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 797.058795] Lustre: lustre-MDT0000: new disk, initializing [ 797.117629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 797.122678] Lustre: Skipped 3 previous similar messages [ 797.136477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 801.738374] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 812.291094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 812.392969] Lustre: 29650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 812.658833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 817.410643] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 827.588951] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 833.923118] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 840.963031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 841.214714] Lustre: 31619:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 843.230421] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 843.296630] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 843.367575] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 843.388232] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 847.371921] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 857.178914] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 865.188393] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 884.200155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 884.202071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 884.207746] LustreError: Skipped 1 previous similar message [ 884.209225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 884.209308] Lustre: Skipped 2 previous similar messages [ 884.240626] Lustre: Skipped 2 previous similar messages [ 890.240958] Lustre: server umount lustre-OST0000 complete [ 900.911186] Lustre: server umount lustre-MDT0000 complete [ 904.674570] LustreError: 29660:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 904.686859] LustreError: 29660:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 904.856965] LustreError: 29640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787076809 with bad export cookie 17111150853244593175 [ 904.859991] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 904.871903] LustreError: 29640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 905.286124] Lustre: server umount lustre-MDT0001 complete [ 917.670056] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 920.618421] Key type lgssc unregistered [ 920.884474] LNet: 33600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.894334] LNetError: 33600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.911763] LNet: Removed LNI 192.168.204.149@tcp [ 921.783390] Key type .llcrypt unregistered [ 921.789884] Key type ._llcrypt unregistered [ 948.976632] Key type ._llcrypt registered [ 948.978557] Key type .llcrypt registered [ 949.069138] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 950.117313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 950.158923] alg: No test for adler32 (adler32-zlib) [ 951.222639] Lustre: Lustre: Build Version: 2.17.57_1_g711d631 [ 951.567814] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 953.264191] Key type lgssc registered [ 954.392920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 961.697033] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 961.709994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 963.244036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 967.254499] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 973.822386] LustreError: 35012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 973.880357] LustreError: 35012:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 976.390314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 976.706041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 982.327569] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 990.368561] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 995.724377] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1000.911415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1001.236273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1002.332759] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1005.915808] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1012.899316] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1028.225773] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 14:15:31 (1787076931) [ 1030.185442] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1031.998686] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1038.820809] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 14:15:42 (1787076942) [ 1052.736694] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 14:15:55 (1787076955) [ 1055.297928] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1067.961293] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1067.968193] Lustre: Skipped 2 previous similar messages [ 1070.160044] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 14:16:13 (1787076973) [ 1072.147842] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1072.156980] Lustre: Skipped 2 previous similar messages [ 1080.364195] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1080.373201] Lustre: Skipped 4 previous similar messages [ 1096.420440] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 14:16:40 (1787077000) [ 1099.982878] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1103.632818] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1112.845644] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 14:16:56 (1787077016) [ 1114.556694] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1114.566456] Lustre: Skipped 6 previous similar messages [ 1127.342104] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 14:17:11 (1787077031) [ 1141.729408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.742821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1141.745292] Lustre: Skipped 1 previous similar message [ 1141.756896] Lustre: Skipped 1 previous similar message [ 1146.856447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1146.860021] Lustre: Skipped 1 previous similar message [ 1151.981931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1152.001468] Lustre: Skipped 1 previous similar message [ 1155.040711] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1155.286362] Lustre: server umount lustre-OST0000 complete [ 1161.186369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1161.193106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1161.214634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1162.212416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1165.851584] Lustre: server umount lustre-MDT0000 complete [ 1167.330249] LustreError: 35956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.355827] LustreError: 35956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1169.728991] LustreError: 34993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787077074 with bad export cookie 17134796633400601696 [ 1169.743771] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1169.752194] LustreError: 34993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1170.110352] Lustre: server umount lustre-MDT0001 complete [ 1180.722864] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 1183.718652] Key type lgssc unregistered [ 1184.001943] LNet: 42880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1184.019715] LNetError: 42880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1184.060684] LNet: Removed LNI 192.168.204.149@tcp [ 1184.807782] Key type .llcrypt unregistered [ 1184.810706] Key type ._llcrypt unregistered [ 1206.315517] Key type ._llcrypt registered [ 1206.318572] Key type .llcrypt registered [ 1206.511910] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 1223.691771] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 1224.843198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1224.863654] alg: No test for adler32 (adler32-zlib) [ 1225.885799] Lustre: Lustre: Build Version: 2.17.57_1_g711d631 [ 1226.088317] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 1227.753798] Key type lgssc registered [ 1228.732894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1264.339821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1264.362133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1265.771617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1265.807370] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1265.926858] Lustre: lustre-MDT0000: new disk, initializing [ 1266.028914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1266.052602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1271.328109] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1281.262243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.449735] Lustre: 46035:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1281.556912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1281.561674] Lustre: Skipped 1 previous similar message [ 1281.624501] Lustre: lustre-MDT0001: new disk, initializing [ 1281.735069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1281.773880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1281.787304] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1286.254977] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1296.065950] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1302.224422] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1309.311477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1309.565344] Lustre: lustre-OST0000: new disk, initializing [ 1309.569321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1309.575112] Lustre: 48004:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1309.641645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1315.896948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1315.909940] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1316.017381] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1316.022591] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1324.716413] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1329.081962] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1329.271737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1333.460546] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1333.676735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1336.804153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.813111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1336.827331] Lustre: Skipped 1 previous similar message [ 1336.828460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1341.902510] Lustre: server umount lustre-OST0000 complete [ 1347.055111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.056873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1347.074681] Lustre: Skipped 1 previous similar message [ 1347.104549] Lustre: Skipped 2 previous similar messages [ 1352.173939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1352.180962] Lustre: Skipped 1 previous similar message [ 1352.837310] Lustre: server umount lustre-MDT0000 complete [ 1356.262703] LustreError: 46044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1356.292272] LustreError: 46044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1356.948942] LustreError: 46026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787077261 with bad export cookie 14548356837032512172 [ 1356.954250] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1356.963605] LustreError: 46026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1357.536574] Lustre: server umount lustre-MDT0001 complete [ 1367.828829] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 14:21:11 (1787077271) [ 1375.388268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1375.868305] LustreError: 50465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1375.984584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1381.126537] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1381.354531] LustreError: 50466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1385.448775] LustreError: 50465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1389.273887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1389.708169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1394.476350] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1403.612958] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1410.127636] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1417.190088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1417.562872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1423.937296] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1431.397364] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1439.675343] LustreError: 52992:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1440.711306] Lustre: Modifying parameter general.timeout=1 in log params [ 1441.767696] Lustre: Modifying parameter general.timeout=2 in log params [ 1444.397883] Lustre: Modifying parameter general.timeout=5 in log params [ 1444.404192] Lustre: Skipped 2 previous similar messages [ 1448.731841] Lustre: Modifying parameter general.timeout=10 in log params [ 1448.736209] Lustre: Skipped 4 previous similar messages [ 1457.526992] Lustre: Modifying parameter general.timeout=20 in log params [ 1457.535427] Lustre: Skipped 9 previous similar messages [ 1473.620527] Lustre: Modifying parameter general.timeout=36 in log params [ 1473.641395] Lustre: Skipped 15 previous similar messages [ 1506.003734] Lustre: Modifying parameter general.timeout=70 in log params [ 1506.009791] Lustre: Skipped 33 previous similar messages [ 1571.383459] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 14:24:35 (1787077475) [ 1572.414964] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1572.422362] Lustre: Skipped 60 previous similar messages [ 1574.346150] Lustre: 59937:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1576.125593] Lustre: 60036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1579.624967] Lustre: 60239:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1579.641916] Lustre: 60239:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1611.758105] Lustre: 61810:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1611.764733] Lustre: 61810:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1611.783148] LustreError: 61810:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1616.064331] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 14:25:19 (1787077519) [ 1630.178063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1630.201304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1630.223568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1635.092525] Lustre: server umount lustre-MDT0000 complete [ 1636.834832] LustreError: 50465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1636.872101] LustreError: 50465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1638.907679] LustreError: 52306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787077543 with bad export cookie 14548356837032513054 [ 1638.911813] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1638.936955] LustreError: 52306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1639.501620] Lustre: server umount lustre-MDT0001 complete [ 1654.142090] Lustre: server umount lustre-OST0000 complete [ 1679.505488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1679.813381] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1680.096342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1683.914307] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1692.412984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1692.591978] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1692.640144] Lustre: 64340:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1697.368243] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1705.713784] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1711.418481] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1719.079573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1719.282942] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1719.744800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1719.750336] Lustre: Skipped 1 previous similar message [ 1725.979952] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1733.050337] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1734.738741] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1734.754787] Lustre: Skipped 22 previous similar messages [ 1734.805232] Lustre: 66905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1742.827839] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1742.848222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1742.870408] Lustre: Skipped 2 previous similar messages [ 1742.889081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1742.903380] Lustre: Skipped 3 previous similar messages [ 1747.701782] Lustre: server umount lustre-OST0000 complete [ 1753.573400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1753.576365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1753.589108] Lustre: Skipped 2 previous similar messages [ 1753.601949] Lustre: Skipped 3 previous similar messages [ 1765.344133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1765.513433] Lustre: server umount lustre-MDT0000 complete [ 1768.875982] LustreError: 66200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787077673 with bad export cookie 14548356837032516155 [ 1768.879284] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1768.888527] LustreError: 66200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1769.278309] Lustre: server umount lustre-MDT0001 complete [ 1779.800916] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 1783.157460] Key type lgssc unregistered [ 1783.566851] LNet: 68118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.578842] LNetError: 68118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.595967] LNet: Removed LNI 192.168.204.149@tcp [ 1784.514793] Key type .llcrypt unregistered [ 1784.517046] Key type ._llcrypt unregistered [ 1810.882652] Key type ._llcrypt registered [ 1810.886285] Key type .llcrypt registered [ 1810.986336] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 1812.281153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1812.320666] alg: No test for adler32 (adler32-zlib) [ 1813.512560] Lustre: Lustre: Build Version: 2.17.57_1_g711d631 [ 1813.848727] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 1815.640193] Key type lgssc registered [ 1817.305065] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1826.282563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1826.306837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1827.746747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1831.960189] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1837.033697] LustreError: 69533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1837.055940] LustreError: 69533:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1840.611724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1841.268637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1845.610539] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1853.014435] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1859.421863] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1866.655937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1867.160567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1873.019294] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1880.321288] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1891.673723] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1901.027228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.033674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1901.061539] Lustre: Skipped 2 previous similar messages [ 1901.083262] Lustre: Skipped 2 previous similar messages [ 1906.149896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1906.161358] Lustre: Skipped 2 previous similar messages [ 1907.432911] Lustre: server umount lustre-MDT0000 complete [ 1911.279354] LustreError: 69528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1911.302925] LustreError: 69528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1911.336302] LustreError: 72219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787077816 with bad export cookie 12060392344980240500 [ 1911.338904] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1911.362882] LustreError: 72219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1911.753050] Lustre: server umount lustre-MDT0001 complete [ 1926.637026] Lustre: server umount lustre-OST0000 complete [ 1956.064299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1956.457434] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1956.487581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1956.754957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1961.132549] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1970.427275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1970.568413] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1970.601125] Lustre: 74408:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1971.028688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1975.378839] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 1983.643521] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1990.145785] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1997.653578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1997.850594] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1998.170827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2004.688269] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2013.407739] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2015.280758] Lustre: 76983:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2017.325563] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2024.931395] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2024.947316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2024.989800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2025.461617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2028.984361] Lustre: server umount lustre-OST0000 complete [ 2035.682676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2035.692716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2035.704103] Lustre: Skipped 2 previous similar messages [ 2035.726912] Lustre: Skipped 3 previous similar messages [ 2039.504948] Lustre: server umount lustre-MDT0000 complete [ 2040.804266] LustreError: 77236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2040.823643] LustreError: 77236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2043.187445] LustreError: 76985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787077948 with bad export cookie 12060392344980241354 [ 2043.199971] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2043.624905] Lustre: server umount lustre-MDT0001 complete [ 2054.323403] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 2057.283995] Key type lgssc unregistered [ 2057.623946] LNet: 78200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2057.634613] LNetError: 78200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2057.663103] LNet: Removed LNI 192.168.204.149@tcp [ 2058.401105] Key type .llcrypt unregistered [ 2058.403773] Key type ._llcrypt unregistered [ 2082.524747] Key type ._llcrypt registered [ 2082.532292] Key type .llcrypt registered [ 2082.609149] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 2083.700905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2083.779492] alg: No test for adler32 (adler32-zlib) [ 2084.883406] Lustre: Lustre: Build Version: 2.17.57_1_g711d631 [ 2085.117360] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 2086.784172] Key type lgssc registered [ 2087.864113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2095.827238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2095.860925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2097.427443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2102.042379] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2107.874094] LustreError: 79615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2107.912947] LustreError: 79615:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2111.140842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2111.654025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2115.670108] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2122.909885] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2128.455618] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2135.301095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2135.774551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2141.861083] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2148.271798] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2154.013062] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2154.682462] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2156.091976] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2156.097282] Lustre: Skipped 1 previous similar message [ 2158.396892] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2158.410868] Lustre: Skipped 2 previous similar messages [ 2163.281177] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2163.288502] Lustre: Skipped 5 previous similar messages [ 2171.882629] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2171.887758] Lustre: Skipped 10 previous similar messages [ 2188.035155] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2188.038618] Lustre: Skipped 21 previous similar messages [ 2220.311198] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2220.315716] Lustre: Skipped 41 previous similar messages [ 2236.441458] Lustre: 87715:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2238.940285] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 14:35:42 (1787078142) [ 2240.231378] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2241.714071] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 14:35:45 (1787078145) [ 2245.604619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2245.608230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2245.620271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2245.633773] Lustre: Skipped 1 previous similar message [ 2250.722616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2250.731344] Lustre: Skipped 1 previous similar message [ 2250.952655] Lustre: server umount lustre-OST0000 complete [ 2254.817468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2254.829983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2254.837964] Lustre: Skipped 1 previous similar message [ 2254.849970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2255.844154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2260.275788] Lustre: server umount lustre-MDT0000 complete [ 2260.967025] LustreError: 85873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2260.985645] LustreError: 85873:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2263.481065] LustreError: 79595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787078168 with bad export cookie 17805106839512012373 [ 2263.482569] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2263.492488] LustreError: 79595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2263.746298] Lustre: server umount lustre-MDT0001 complete [ 2273.251303] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 2275.938914] Key type lgssc unregistered [ 2276.194096] LNet: 88966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2276.202659] LNetError: 88966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2276.232893] LNet: Removed LNI 192.168.204.149@tcp [ 2276.911114] Key type .llcrypt unregistered [ 2276.914697] Key type ._llcrypt unregistered [ 2280.717567] Key type ._llcrypt registered [ 2280.720617] Key type .llcrypt registered [ 2280.834279] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2284.729683] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules [ 2285.701690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2285.722119] alg: No test for adler32 (adler32-zlib) [ 2286.711835] Lustre: Lustre: Build Version: 2.17.57_1_g711d631 [ 2286.919663] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 2288.576685] Key type lgssc registered [ 2289.584499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2290.685578] LustreError: 90169:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2306.413171] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 2312.565249] LustreError: 90169:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2314.551904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2314.570672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2316.058539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2320.294651] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2325.482664] LustreError: 91192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2325.495127] LustreError: 91192:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2327.257498] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 14:37:11 (1787078231) [ 2330.597562] LustreError: 91193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2330.846209] LustreError: 91224:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2330.852343] LustreError: 91224:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2330.856682] LustreError: 91224:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 2331.115645] Lustre: server umount lustre-MDT0000 complete [ 2342.125436] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 2344.509132] Key type lgssc unregistered [ 2344.807907] LNet: 92720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2344.812123] LNetError: 92720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2344.838906] LNet: Removed LNI 192.168.204.149@tcp [ 2345.504703] Key type .llcrypt unregistered [ 2345.514183] Key type ._llcrypt unregistered [ 2359.472618] Key type ._llcrypt registered [ 2359.475851] Key type .llcrypt registered [ 2359.545972] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 2360.176959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2360.226841] alg: No test for adler32 (adler32-zlib) [ 2361.243195] Lustre: Lustre: Build Version: 2.17.57_1_g711d631 [ 2361.423598] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 2363.049780] Key type lgssc registered [ 2363.999141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2370.227572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2370.239758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2371.701597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2375.591406] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2381.284860] LustreError: 93918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.300980] LustreError: 93918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2382.721684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2383.193708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2386.967108] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2393.190314] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2397.774580] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2403.537353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2404.080641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2409.648916] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2415.981613] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2457.631703] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 14:39:21 (1787078361) [ 2459.287945] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2460.739816] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 14:39:24 (1787078364) [ 2464.737782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2464.747309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2464.772648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2467.819130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2467.821949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2470.045533] Lustre: server umount lustre-MDT0000 complete [ 2472.932078] LustreError: 93914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2472.954404] LustreError: 93914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2473.236822] LustreError: 93899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787078378 with bad export cookie 3967304490205816255 [ 2473.241424] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2473.250150] LustreError: 93899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2473.601283] Lustre: server umount lustre-MDT0001 complete [ 2486.395358] Lustre: server umount lustre-OST0000 complete [ 2495.570512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2495.966788] LustreError: 98188:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.100602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2499.609843] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2501.611262] LustreError: 98188:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2501.626487] LustreError: 98188:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2505.700894] LustreError: 98189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.118715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2506.582956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2509.868203] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2515.115185] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2519.037841] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2528.091696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2528.225812] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2528.230838] LustreError: 98175:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2528.246776] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2528.253768] LustreError: 100270:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2528.259284] LustreError: 100270:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2528.264164] LustreError: 100270:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2528.314660] Lustre: server umount lustre-OST0000 complete [ 2528.317267] LustreError: 100270:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2533.584711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2533.702635] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2533.710700] LustreError: 98177:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2533.718603] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2533.725918] LustreError: 100731:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2533.731418] LustreError: 100731:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2533.735638] LustreError: 100731:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2533.787431] Lustre: server umount lustre-OST0000 complete [ 2533.791252] LustreError: 100731:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2540.703451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2540.855801] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2540.862356] Lustre: Found index 0 for lustre-OST0000, updating log [ 2540.868417] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2540.923327] Lustre: lustre-OST0000: new disk, initializing [ 2540.927510] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2540.933790] Lustre: 101266:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2541.148764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2545.120583] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2547.291428] Lustre: Failing over lustre-OST0000 [ 2547.385385] Lustre: server umount lustre-OST0000 complete [ 2551.265890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2551.267634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2551.276901] Lustre: Skipped 2 previous similar messages [ 2551.283934] Lustre: Skipped 3 previous similar messages [ 2555.721453] Lustre: server umount lustre-MDT0000 complete [ 2557.410722] LustreError: 98188:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2557.428585] LustreError: 98188:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2557.979637] LustreError: 100289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787078462 with bad export cookie 3967304490205818999 [ 2557.984052] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2557.992610] LustreError: 100289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2558.165943] Lustre: server umount lustre-MDT0001 complete [ 2563.114512] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 14:41:07 (1787078467) [ 2572.692530] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 2578.857889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2579.137710] LustreError: 103603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.228513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2581.795817] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2586.938234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2590.478580] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2592.425301] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2596.348754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2600.025217] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2605.005845] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2605.068721] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2605.153079] Lustre: lustre-OST0001: new disk, initializing [ 2605.157159] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2605.161067] Lustre: 105922:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2608.836990] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2611.253737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2611.261495] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2611.329780] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2614.159843] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2618.194823] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2620.806397] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.49@tcp (stopping) [ 2620.812537] Lustre: Skipped 2 previous similar messages [ 2621.922507] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2621.936494] Lustre: Skipped 2 previous similar messages [ 2625.902836] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.49@tcp (stopping) [ 2625.908593] Lustre: Skipped 3 previous similar messages [ 2626.802139] Lustre: server umount lustre-MDT0001 complete [ 2627.048425] LustreError: 103598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.059599] LustreError: 103598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2632.074311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2632.146536] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2632.149586] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2632.183328] Lustre: 103591:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2632.545775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2632.548637] Lustre: Skipped 3 previous similar messages [ 2633.770964] LustreError: 107575:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2633.777868] LustreError: 107575:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2633.783532] LustreError: 103595:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2633.827880] LustreError: 107590:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.149@tcp: cfg command failed: rc = -17 [ 2633.833686] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2633.833686] [ 2634.482570] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2637.807624] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2638.648668] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 14:42:23 (1787078543) [ 2646.528480] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 2655.466821] Lustre: 109530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2667.636304] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2669.610912] Lustre: 110664:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2735.072649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2735.074676] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2735.076834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2735.110808] Lustre: Skipped 2 previous similar messages [ 2739.091715] Lustre: server umount lustre-MDT0000 complete [ 2740.193044] LustreError: 104503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2740.204569] LustreError: 104503:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2741.383816] LustreError: 103585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787078646 with bad export cookie 3967304490205819881 [ 2741.388098] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2741.394599] LustreError: 103585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2741.605550] Lustre: server umount lustre-MDT0001 complete [ 2754.625676] Lustre: server umount lustre-OST0000 complete [ 2766.160839] Lustre: server umount lustre-OST0001 complete [ 2769.609771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2775.761411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2780.708637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2786.624960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2798.335930] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 2805.231235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2805.244621] Lustre: lustre-MDT0000: reset Object Index mappings [ 2805.718854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2808.880691] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2814.568542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2814.582921] Lustre: lustre-MDT0001: reset Object Index mappings [ 2817.819925] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2819.608915] Lustre: 115760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2823.721120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2825.197324] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2827.927822] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2831.341323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2832.659224] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2836.434874] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2838.001965] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2838.004286] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2841.522443] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2843.901847] Lustre: 117634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2877.016279] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 14:46:21 (1787078781) [ 2907.106357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2907.112069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2907.124880] Lustre: Skipped 1 previous similar message [ 2907.127899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2907.132178] Lustre: Skipped 3 previous similar messages [ 2911.397444] Lustre: server umount lustre-MDT0000 complete [ 2913.738832] LustreError: 114594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787078818 with bad export cookie 3967304490205866753 [ 2913.752834] LustreError: 114594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2913.754403] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2914.788651] LustreError: 116701:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2914.805096] LustreError: 116701:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2928.096218] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2928.246845] Lustre: server umount lustre-MDT0001 complete [ 2936.481946] Lustre: server umount lustre-OST0000 complete [ 2953.184122] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2953.311541] Lustre: server umount lustre-OST0001 complete [ 2957.619943] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 2964.103953] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 2985.999764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2986.154232] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2986.179924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2986.247907] Lustre: lustre-MDT0000: new disk, initializing [ 2986.300052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2986.304791] Lustre: Skipped 3 previous similar messages [ 2986.318352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2988.654325] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 2993.188782] Lustre: server umount lustre-MDT0000 complete [ 2999.220246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2999.413282] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2999.424480] Lustre: 121903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2999.438443] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3002.228736] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3005.457198] Lustre: server umount lustre-MDT0000 complete [ 3010.759233] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 14:48:35 (1787078915) [ 3011.615995] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3012.711779] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 14:48:37 (1787078917) [ 3022.712481] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 3027.944258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3030.389935] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3035.106092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3035.181129] Lustre: 123949:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3035.219592] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3035.222645] Lustre: Skipped 1 previous similar message [ 3035.273585] Lustre: lustre-MDT0001: new disk, initializing [ 3035.368867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3035.379179] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3037.749056] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3040.912691] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3044.943935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3045.091327] Lustre: lustre-OST0000: new disk, initializing [ 3045.094970] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3045.099782] Lustre: 125526:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3046.825662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3046.833077] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3046.925892] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3048.726278] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3055.300691] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3055.406583] Lustre: lustre-OST0001: new disk, initializing [ 3055.416243] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3055.420501] Lustre: 126398:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3057.205420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3057.215601] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3057.304181] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3059.323294] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3065.663996] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3067.608906] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3330.951327] Lustre: 132793:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3330.956717] Lustre: 132793:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3330.959735] Lustre: 132793:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3330.962987] Lustre: 132793:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3330.967418] Lustre: 132793:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3330.971442] Lustre: 132793:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3552.269740] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:57:35 (1787079455) [ 3568.101020] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3568.112342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3568.122115] Lustre: Skipped 5 previous similar messages [ 3568.125976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3568.132332] Lustre: Skipped 10 previous similar messages [ 3572.846871] Lustre: server umount lustre-MDT0000 complete [ 3574.758251] LustreError: 123957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3574.773925] LustreError: 123957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3576.459984] LustreError: 127161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787079481 with bad export cookie 3967304490205930173 [ 3576.477105] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3576.478386] LustreError: 127161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3579.882152] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3579.884939] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3579.899120] Lustre: Skipped 4 previous similar messages [ 3579.915025] Lustre: Skipped 5 previous similar messages [ 3582.873977] Lustre: server umount lustre-MDT0001 complete [ 3592.336078] Lustre: server umount lustre-OST0000 complete [ 3601.542719] Lustre: server umount lustre-OST0001 complete [ 3607.216685] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 3614.466672] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 3642.769824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3642.939888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3642.970453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3643.103957] Lustre: lustre-MDT0000: new disk, initializing [ 3643.228296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3643.236370] Lustre: Skipped 5 previous similar messages [ 3643.247957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3647.051046] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3655.630854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3655.732631] Lustre: 135834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3656.067283] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3660.398787] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3668.534311] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3673.258052] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3678.777338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3679.078749] Lustre: lustre-OST0000: new disk, initializing [ 3679.081743] Lustre: Skipped 1 previous similar message [ 3679.086525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3679.091773] Lustre: Skipped 2 previous similar messages [ 3679.095773] Lustre: 137804:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3681.031746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3681.039743] Lustre: Skipped 1 previous similar message [ 3681.042655] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3681.148576] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3684.269866] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 3690.820418] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3697.210956] Lustre: lustre-MDD0000: changelog on [ 3699.134525] Lustre: lustre-MDD0001: changelog on [ 5076.789401] Lustre: lustre-MDD0001: changelog off [ 5077.830769] Lustre: 141667:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5078.306743] Lustre: lustre-MDD0000: changelog off [ 5084.131605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5084.133462] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5084.141239] LustreError: Skipped 1 previous similar message [ 5084.152359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5084.162317] Lustre: Skipped 3 previous similar messages [ 5087.433751] Lustre: server umount lustre-OST0000 complete [ 5089.763756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5089.768142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5089.774053] Lustre: Skipped 1 previous similar message [ 5089.778579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5094.370834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5094.372668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5095.779378] Lustre: server umount lustre-MDT0000 complete [ 5097.785066] LustreError: 135828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787081002 with bad export cookie 3967304490206741921 [ 5097.787255] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5097.790863] LustreError: 135828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5098.001348] Lustre: server umount lustre-MDT0001 complete [ 5103.295751] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 5104.799892] Key type lgssc unregistered [ 5104.982825] LNet: 146864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5104.989170] LNetError: 146864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5104.999763] LNet: Removed LNI 192.168.204.149@tcp [ 5105.486229] Key type .llcrypt unregistered [ 5105.487915] Key type ._llcrypt unregistered [ 5115.915476] Key type ._llcrypt registered [ 5115.917417] Key type .llcrypt registered [ 5115.974785] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 5124.930609] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 5125.570451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5125.582465] alg: No test for adler32 (adler32-zlib) [ 5126.499997] Lustre: Lustre: Build Version: 2.17.57_1_g711d631 [ 5126.658169] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 5128.280515] Key type lgssc registered [ 5129.017583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5147.602382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5147.614063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5148.728308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5148.746842] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5148.795336] Lustre: lustre-MDT0000: new disk, initializing [ 5148.834768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5148.843493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5150.866697] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5156.923847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5156.987519] Lustre: 150042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5157.034853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5157.041376] Lustre: Skipped 1 previous similar message [ 5157.100926] Lustre: lustre-MDT0001: new disk, initializing [ 5157.170413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5157.190929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5157.199707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5159.619529] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5164.870402] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5167.996480] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5171.546082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5171.719859] Lustre: lustre-OST0000: new disk, initializing [ 5171.722700] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5171.727409] Lustre: 152008:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5171.777361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5172.296591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5172.301570] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5172.360405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5174.800902] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5180.039768] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5184.043099] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_module obdecho/obdecho [ 5184.999191] LustreError: 153009:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5185.004844] LustreError: 153009:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5185.901052] LustreError: 153106:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5193.186915] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5193.191577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5193.194995] Lustre: Skipped 2 previous similar messages [ 5193.195436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5193.205357] Lustre: Skipped 2 previous similar messages [ 5197.195483] Lustre: server umount lustre-MDT0000 complete [ 5199.079086] LustreError: 150034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787081104 with bad export cookie 13460780252792063668 [ 5199.082817] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5199.084463] LustreError: 150034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5199.256593] Lustre: server umount lustre-MDT0001 complete [ 5211.650401] Lustre: server umount lustre-OST0000 complete [ 5221.946271] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 5227.084189] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 5244.380597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5244.515319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5244.536353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5244.585910] Lustre: lustre-MDT0000: new disk, initializing [ 5244.638578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5244.655947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5246.503171] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5252.243169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5252.315732] Lustre: 156815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5252.364056] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5252.368658] Lustre: Skipped 1 previous similar message [ 5252.415574] Lustre: lustre-MDT0001: new disk, initializing [ 5252.463214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5252.490606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5252.500079] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5254.757991] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5259.859687] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5262.537669] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5265.653693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5265.784630] Lustre: lustre-OST0000: new disk, initializing [ 5265.787654] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5265.793286] Lustre: 158783:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5265.853369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5268.686987] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5272.093439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5272.102811] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5272.186601] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5274.118732] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5277.308166] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5277.500644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5282.211849] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5282.556604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5287.400728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5287.401804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5287.422354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5290.961184] Lustre: server umount lustre-OST0000 complete [ 5297.636974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5297.644312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5297.662232] Lustre: Skipped 2 previous similar messages [ 5297.680781] Lustre: Skipped 2 previous similar messages [ 5302.737923] Lustre: server umount lustre-MDT0000 complete [ 5302.779137] LustreError: 156826:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5302.811249] LustreError: 156826:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5307.884862] LustreError: 156827:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5307.924883] LustreError: 156827:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5311.317578] LustreError: 156806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787081216 with bad export cookie 13460780252792064620 [ 5311.321415] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5311.323923] LustreError: 156806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5311.877325] Lustre: server umount lustre-MDT0001 complete [ 5319.565397] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 15:27:01 (1787081221) [ 5323.296310] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5327.528397] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 15:27:09 (1787081229) [ 5341.129671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5341.563649] LustreError: 161195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5341.663385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5346.795767] LustreError: 161196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5346.940148] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5351.910713] LustreError: 161195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5356.440787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5361.699904] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5369.036144] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5374.882871] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5381.480350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5382.083514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5382.094647] Lustre: Skipped 1 previous similar message [ 5388.530795] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5394.927678] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5411.813617] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5411.822767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5411.841406] Lustre: Skipped 2 previous similar messages [ 5411.858679] Lustre: Skipped 2 previous similar messages [ 5415.826479] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.49@tcp (stopping) [ 5417.464690] Lustre: server umount lustre-MDT0000 complete [ 5420.915491] LustreError: 161192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5421.077867] LustreError: 163040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787081326 with bad export cookie 13460780252792065495 [ 5421.084857] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5421.090533] LustreError: 163040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5422.055032] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5426.074740] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.49@tcp (stopping) [ 5426.084376] Lustre: Skipped 4 previous similar messages [ 5427.529496] Lustre: server umount lustre-MDT0001 complete [ 5433.370527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5444.392677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5444.590959] LustreError: 165062:0:(llog.c:1655:llog_backup()) MGC192.168.204.149@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5444.600049] Lustre: 165062:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.149@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5446.380797] LustreError: 165067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5446.407888] LustreError: 165067:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5481.442926] LustreError: 165069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5481.456076] LustreError: 165069:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [ 5506.016701] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5506.030531] Lustre: 165062:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.149@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5546.987960] LustreError: 165067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5546.993284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5546.997086] LustreError: 165067:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 42 previous similar messages [ 5548.014161] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5550.354208] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5557.330394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5567.457470] LustreError: 148266:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007edd0a90 x1873890279281792/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5570.864271] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5573.097075] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5577.131201] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5578.553251] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5583.274177] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5584.686615] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5600.744985] Lustre: server umount lustre-MDT0000 complete [ 5603.615279] LustreError: 161175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787081508 with bad export cookie 13460780252792071949 [ 5603.622329] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5603.627676] LustreError: 161175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5603.810405] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5603.824198] Lustre: Skipped 1 previous similar message [ 5603.892280] Lustre: server umount lustre-MDT0001 complete [ 5606.998480] Lustre: server umount lustre-OST0000 complete [ 5624.100657] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 5631.616582] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 5660.505249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5660.700766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5660.722843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5660.780557] Lustre: lustre-MDT0000: new disk, initializing [ 5660.844254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5664.712182] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5674.196330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5674.306409] Lustre: 170159:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5674.525740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5678.127668] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5685.178115] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5688.689582] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5692.883410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5693.060370] Lustre: lustre-OST0000: new disk, initializing [ 5693.063354] Lustre: Skipped 1 previous similar message [ 5693.065884] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5693.069519] Lustre: Skipped 2 previous similar messages [ 5693.076818] Lustre: 172125:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5693.147426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5693.152806] Lustre: Skipped 3 previous similar messages [ 5696.705943] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5699.141429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5699.149333] Lustre: Skipped 1 previous similar message [ 5699.154041] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5699.257212] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5703.467967] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5707.188468] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5707.388650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5710.801786] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5710.945398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5714.915286] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5714.915410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5714.921773] LustreError: Skipped 1 previous similar message [ 5714.923145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5714.929200] Lustre: Skipped 1 previous similar message [ 5718.590609] Lustre: server umount lustre-OST0000 complete [ 5735.392383] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5735.539723] Lustre: server umount lustre-MDT0000 complete [ 5738.921182] LustreError: 170952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787081643 with bad export cookie 13460780252792088784 [ 5738.923746] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5738.932602] LustreError: 170952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5739.442431] Lustre: server umount lustre-MDT0001 complete [ 5754.335941] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 15:34:16 (1787081656) [ 5767.700482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5769.011951] LustreError: 174540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5769.034324] LustreError: 174540:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 22 previous similar messages [ 5775.783478] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5784.846340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5789.664160] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5797.583756] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5802.815684] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5809.007470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5817.148930] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5825.555937] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5835.332961] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 15:35:38 (1787081738) [ 5840.358350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5840.374746] Lustre: Skipped 2 previous similar messages [ 5840.383630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5840.388111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5840.418676] Lustre: Skipped 8 previous similar messages [ 5845.606332] Lustre: server umount lustre-OST0000 complete [ 5855.883715] Lustre: server umount lustre-MDT0000 complete [ 5860.143449] LustreError: 174520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787081765 with bad export cookie 13460780252792089659 [ 5860.147308] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5860.153842] LustreError: 174520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5860.571424] Lustre: server umount lustre-MDT0001 complete [ 5871.931069] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 5874.599779] Key type lgssc unregistered [ 5874.917538] LNet: 178273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5874.932362] LNetError: 178273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5874.957651] LNet: Removed LNI 192.168.204.149@tcp [ 5875.866048] Key type .llcrypt unregistered [ 5875.868731] Key type ._llcrypt unregistered [ 5894.236966] Key type ._llcrypt registered [ 5894.246071] Key type .llcrypt registered [ 5894.367395] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 5895.366173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5895.490864] alg: No test for adler32 (adler32-zlib) [ 5896.594522] Lustre: Lustre: Build Version: 2.17.57_1_g711d631 [ 5896.844142] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 5898.528164] Key type lgssc registered [ 5899.518256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5906.878592] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5906.891780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5922.336739] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5927.456555] LustreError: 179516:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.149@tcp: failed processing log, type 4: rc = -110 [ 5932.576813] LustreError: 179516:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5932.609684] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5932.629205] LustreError: MGC192.168.204.149@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5932.649581] LustreError: 179516:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5932.667172] LustreError: 179516:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 5932.684459] LustreError: 179516:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5932.836295] Lustre: server umount lustre-OST0000 complete [ 5932.841328] LustreError: 179516:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5942.581481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5944.045291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5947.616768] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5954.529393] LustreError: 180015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5954.550714] LustreError: 180015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5955.540455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5955.785771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5959.367724] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5966.533876] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5971.045577] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5976.652434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5976.847788] LustreError: 181831:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5976.856906] Lustre: 181831:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.149@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5977.063921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5982.830661] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 5989.324754] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5991.907697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5991.915248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5991.938200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5994.466029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5994.466928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5998.021822] Lustre: server umount lustre-MDT0000 complete [ 5999.591025] LustreError: 181094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5999.607340] LustreError: 181094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6001.671195] LustreError: 179995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787081906 with bad export cookie 5801621389329398173 [ 6001.673632] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6001.675869] LustreError: 179995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6001.970364] Lustre: server umount lustre-MDT0001 complete [ 6017.458278] Lustre: server umount lustre-OST0000 complete [ 6028.234074] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 6039.291730] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 6079.923292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6080.138248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6080.164774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6080.244635] Lustre: lustre-MDT0000: new disk, initializing [ 6080.328391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6080.361594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6084.795981] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6095.317269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6095.448390] Lustre: 185358:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6095.516635] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6095.520576] Lustre: Skipped 1 previous similar message [ 6095.604365] Lustre: lustre-MDT0001: new disk, initializing [ 6095.687778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6095.730089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6095.742660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6100.612972] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6110.283333] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6116.281984] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6123.523995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6123.779591] Lustre: lustre-OST0000: new disk, initializing [ 6123.786116] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6123.794812] Lustre: 187327:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6123.909735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6131.032866] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6131.300744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6131.316653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6131.490751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6140.739550] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6146.645969] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6146.890352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6152.926627] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6153.235393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6157.290885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6157.295162] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6157.302075] Lustre: Skipped 1 previous similar message [ 6157.316216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6157.340679] Lustre: Skipped 3 previous similar messages [ 6162.415168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6162.427760] Lustre: Skipped 1 previous similar message [ 6167.530408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6167.539038] Lustre: Skipped 1 previous similar message [ 6170.080320] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6170.274113] Lustre: server umount lustre-OST0000 complete [ 6177.765365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6177.774848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6177.794953] Lustre: Skipped 1 previous similar message [ 6177.805635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6177.817634] LustreError: Skipped 1 previous similar message [ 6182.889847] LustreError: 187809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6183.018511] Lustre: server umount lustre-MDT0000 complete [ 6188.032632] LustreError: 185370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6188.064464] LustreError: 185370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6189.558766] LustreError: 187325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787082094 with bad export cookie 5801621389329398887 [ 6189.592193] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6189.611155] LustreError: 187325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6190.033704] Lustre: server umount lustre-MDT0001 complete [ 6204.022510] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 15:41:47 (1787082107) [ 6223.399233] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 6226.531068] Key type lgssc unregistered [ 6226.824174] LNet: 190280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6226.831952] LNetError: 190280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6227.885630] LNet: Removed LNI 192.168.204.149@tcp [ 6228.936848] Key type .llcrypt unregistered [ 6228.944994] Key type ._llcrypt unregistered [ 6250.661917] Key type ._llcrypt registered [ 6250.665031] Key type .llcrypt registered [ 6250.830552] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 6252.242236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6252.273425] alg: No test for adler32 (adler32-zlib) [ 6253.440600] Lustre: Lustre: Build Version: 2.17.57_1_g711d631 [ 6253.802725] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 6255.576169] Key type lgssc registered [ 6257.331510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6267.851512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6267.891230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6269.397812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6274.897497] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6279.655392] LustreError: 191501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6279.677756] LustreError: 191501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6284.771137] LustreError: 191500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6285.557895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6286.150848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6291.077459] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6300.226378] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6307.213464] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6316.600856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6316.763171] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6316.766387] LustreError: 193366:0:(llog.c:1696:llog_backup()) MGC192.168.204.149@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6316.775625] Lustre: 193366:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.149@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6316.922148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6324.232828] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6332.457696] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6335.465829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6335.473756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6335.484748] Lustre: Skipped 2 previous similar messages [ 6335.499965] Lustre: Skipped 2 previous similar messages [ 6340.581467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6340.594750] Lustre: Skipped 2 previous similar messages [ 6345.701746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6345.711144] Lustre: Skipped 2 previous similar messages [ 6349.282997] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6349.575270] Lustre: server umount lustre-MDT0000 complete [ 6350.822811] LustreError: 191495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6350.860099] LustreError: 191495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6353.753332] LustreError: 191481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787082258 with bad export cookie 9680281819567839387 [ 6353.773063] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6354.285873] Lustre: server umount lustre-MDT0001 complete [ 6368.390968] Lustre: server umount lustre-OST0000 complete [ 6378.798916] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 6390.022708] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 6432.722969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6432.977667] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6433.023632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6433.068221] Lustre: lustre-MDT0000: new disk, initializing [ 6433.140955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6433.170411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6439.379972] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6451.480494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6451.628440] Lustre: 196896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6451.704297] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6451.709697] Lustre: Skipped 1 previous similar message [ 6451.715402] LustreError: 197678:0:(llog.c:1696:llog_backup()) MGC192.168.204.149@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6451.722235] LustreError: 197678:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 6451.729613] Lustre: 197678:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.149@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6451.750637] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6451.755159] Lustre: Skipped 1 previous similar message [ 6451.913256] Lustre: lustre-MDT0001: new disk, initializing [ 6452.039669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6452.084360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6452.093034] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6457.074667] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6467.039311] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6475.220796] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6484.226288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6484.744991] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6484.748989] Lustre: Skipped 1 previous similar message [ 6484.752800] LustreError: 198845:0:(llog.c:1696:llog_backup()) MGC192.168.204.149@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6484.759790] LustreError: 198845:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 6484.765652] Lustre: 198845:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.149@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 6484.774550] Lustre: 198845:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6484.802298] Lustre: lustre-OST0000: new disk, initializing [ 6484.812758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6484.836839] Lustre: 198868:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6485.000769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6492.717497] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6494.794818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6494.800831] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6494.990686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6503.649814] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6509.363763] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6509.626610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6514.605655] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6514.847073] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6520.809563] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6520.810361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6520.818700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6520.829398] LustreError: Skipped 1 previous similar message [ 6520.886160] Lustre: Skipped 1 previous similar message [ 6523.309128] Lustre: server umount lustre-OST0000 complete [ 6528.993951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6529.003235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6529.034258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6529.036811] Lustre: Skipped 1 previous similar message [ 6531.041488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6533.782046] Lustre: server umount lustre-MDT0000 complete [ 6536.161322] LustreError: 196903:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6536.191625] LustreError: 196903:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6538.650754] LustreError: 196887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787082443 with bad export cookie 9680281819567840101 [ 6538.670375] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6538.680288] LustreError: 196887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6539.076191] Lustre: server umount lustre-MDT0001 complete [ 6549.054275] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 15:47:32 (1787082452) [ 6569.809257] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 6580.888689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6581.335198] LustreError: 201996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6581.400223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6586.319548] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6591.975945] LustreError: 201997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6592.004906] LustreError: 201997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6595.267506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6599.718725] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6602.948682] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6609.744879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6615.898866] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6625.182257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6625.361284] Lustre: lustre-OST0001: new disk, initializing [ 6625.367629] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6625.378151] Lustre: 204302:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6625.466563] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6625.473680] Lustre: Skipped 2 previous similar messages [ 6631.476827] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6635.094488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6635.116036] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6635.196449] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6642.779523] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6649.257607] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6651.755170] Lustre: DEBUG MARKER: ADD OST3 [ 6656.691620] loop: module loaded [ 6656.803153] loop0: detected capacity change from 0 to 204800000 [ 6657.036881] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6657.053621] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6657.065764] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6663.521602] loop0: detected capacity change from 0 to 204800000 [ 6663.952151] Lustre: DEBUG MARKER: STOP OST3 [ 6668.889309] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6669.052278] Lustre: lustre-OST0003: new disk, initializing [ 6669.057739] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 6669.063232] Lustre: 206304:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6670.798978] Lustre: *** cfs_fail_loc=2109, val=2*** [ 6670.963901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 6670.977913] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 6671.024165] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 6672.864820] LustreError: 206480:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 6673.298709] Lustre: Failing over lustre-OST0003 [ 6673.782849] Lustre: server umount lustre-OST0003 complete [ 6675.938736] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6675.949372] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6675.986370] LustreError: 204830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6676.013589] LustreError: 204830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6677.546718] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6684.847919] Lustre: DEBUG MARKER: START OST3 again [ 6691.796281] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6692.112391] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 6692.121609] Lustre: Skipped 1 previous similar message [ 6692.135673] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 6694.095927] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6694.162584] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 6694.162877] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6698.629845] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6719.510627] Lustre: Failing over lustre-OST0003 [ 6719.647300] Lustre: server umount lustre-OST0003 complete [ 6719.971436] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6719.979712] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6719.994744] Lustre: Skipped 1 previous similar message [ 6720.000077] LustreError: 204830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6720.026455] LustreError: 204830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 6738.403859] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6738.405683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6738.415628] Lustre: Skipped 3 previous similar messages [ 6738.419084] Lustre: Skipped 3 previous similar messages [ 6744.105767] Lustre: server umount lustre-MDT0000 complete [ 6748.105170] LustreError: 201977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787082653 with bad export cookie 9680281819567840983 [ 6748.110842] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6748.114858] LustreError: 201977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6754.619525] Lustre: server umount lustre-MDT0001 complete [ 6759.150949] Lustre: server umount lustre-OST0000 complete [ 6763.496694] Lustre: server umount lustre-OST0001 complete [ 6770.665723] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 6778.866642] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 6815.541682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6815.751914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6815.776817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6815.892097] Lustre: lustre-MDT0000: new disk, initializing [ 6816.012784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6821.051532] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6832.589039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6832.732516] Lustre: 211776:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6832.798728] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6832.803089] Lustre: Skipped 1 previous similar message [ 6832.893591] Lustre: lustre-MDT0001: new disk, initializing [ 6833.038720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6833.044327] Lustre: Skipped 1 previous similar message [ 6833.081280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6833.112219] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6838.775086] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6849.520735] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6856.231416] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6864.244261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6864.681975] Lustre: 213746:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6866.070115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6866.085685] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6866.281368] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6871.632266] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 6881.297871] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6886.221235] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6886.425640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6891.123969] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6891.359177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6897.124639] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6897.125366] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6897.137858] LustreError: Skipped 2 previous similar messages [ 6897.139577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6897.139583] Lustre: Skipped 10 previous similar messages [ 6897.169083] Lustre: Skipped 4 previous similar messages [ 6907.362192] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6907.522448] Lustre: server umount lustre-OST0000 complete [ 6917.603557] LustreError: 211789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6917.629327] LustreError: 211789:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 6917.797421] Lustre: server umount lustre-MDT0000 complete [ 6921.970592] LustreError: 213745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787082826 with bad export cookie 9680281819567843279 [ 6921.978698] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6921.983378] LustreError: 213745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6922.568506] Lustre: server umount lustre-MDT0001 complete [ 6926.569190] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 15:53:50 (1787082830) [ 6942.824480] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 6955.487833] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 6999.345912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6999.656269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6999.704819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6999.709228] Lustre: Skipped 1 previous similar message [ 6999.818116] Lustre: lustre-MDT0000: new disk, initializing [ 6999.827121] Lustre: Skipped 1 previous similar message [ 6999.919339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7006.159145] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7018.865993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7019.015972] Lustre: 218361:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7019.463259] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7024.645837] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7034.619841] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7040.532634] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7047.929909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7048.311884] Lustre: 220333:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7049.761953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7054.812552] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7064.627252] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7068.260472] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7068.463744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7072.452683] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7072.639610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7075.299299] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7075.303616] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7075.324074] Lustre: Skipped 2 previous similar messages [ 7075.332657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7075.343406] Lustre: Skipped 8 previous similar messages [ 7080.771169] Lustre: server umount lustre-OST0000 complete [ 7090.793862] Lustre: server umount lustre-MDT0000 complete [ 7094.574739] LustreError: 218351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787082999 with bad export cookie 9680281819567844161 [ 7094.579496] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7094.584089] LustreError: 218351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7094.925466] Lustre: server umount lustre-MDT0001 complete [ 7103.908192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7104.379346] LustreError: 222549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7104.401342] LustreError: 222549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7104.548773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7104.568166] Lustre: Skipped 4 previous similar messages [ 7110.014170] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7120.198947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7125.424586] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7134.601361] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7141.202346] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7148.864898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7156.727264] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7165.802722] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7176.164182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7176.170772] LustreError: Skipped 1 previous similar message [ 7176.183901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7176.192882] Lustre: Skipped 6 previous similar messages [ 7182.147444] Lustre: server umount lustre-OST0000 complete [ 7192.374188] Lustre: server umount lustre-MDT0000 complete [ 7196.466804] LustreError: 222529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787083101 with bad export cookie 9680281819567845043 [ 7196.476244] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7196.482230] LustreError: 222529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7196.907857] Lustre: server umount lustre-MDT0001 complete [ 7303.005972] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 7305.842831] Key type lgssc unregistered [ 7306.182887] LNet: 226215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7306.207386] LNetError: 226215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7306.225211] LNet: Removed LNI 192.168.204.149@tcp [ 7307.363211] Key type .llcrypt unregistered [ 7307.368399] Key type ._llcrypt unregistered [ 7331.387578] Key type ._llcrypt registered [ 7331.395378] Key type .llcrypt registered [ 7331.525816] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 7346.890710] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 7347.820436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7347.833092] alg: No test for adler32 (adler32-zlib) [ 7349.023118] Lustre: Lustre: Build Version: 2.17.57_1_g711d631 [ 7349.254618] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 7350.960237] Key type lgssc registered [ 7352.067688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7387.221795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7387.262130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7388.647336] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7388.675982] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7388.734843] Lustre: lustre-MDT0000: new disk, initializing [ 7388.827890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7388.849596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7392.965570] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7403.627051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7403.718784] Lustre: 229594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7403.780486] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7403.787455] Lustre: Skipped 1 previous similar message [ 7403.835741] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7403.855521] Lustre: lustre-MDT0001: new disk, initializing [ 7403.958906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7404.011445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7404.019679] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7408.159360] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7417.019662] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7421.876495] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7428.844252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7429.090731] Lustre: lustre-OST0000: new disk, initializing [ 7429.095783] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7429.107259] Lustre: 231561:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7429.213752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7434.220596] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7436.895700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7436.904029] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7437.058491] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7442.651605] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7446.701468] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7446.900571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7450.965548] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7451.201711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7457.764838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7457.771084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7457.791937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7457.798459] Lustre: Skipped 1 previous similar message [ 7459.011682] Lustre: server umount lustre-OST0000 complete [ 7462.881302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7462.890497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7462.903669] Lustre: Skipped 2 previous similar messages [ 7462.912673] Lustre: Skipped 1 previous similar message [ 7468.002112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7468.014702] Lustre: Skipped 1 previous similar message [ 7473.121636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7473.131175] Lustre: Skipped 1 previous similar message [ 7476.192165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7476.361559] Lustre: server umount lustre-MDT0000 complete [ 7479.818815] LustreError: 229585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787083384 with bad export cookie 2209941076095280969 [ 7479.824944] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7479.836239] LustreError: 229585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7480.152565] Lustre: server umount lustre-MDT0001 complete [ 7498.002240] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 7507.474062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7507.900338] LustreError: 234497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7507.929762] LustreError: 234497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7507.972877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7512.472548] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7513.075282] LustreError: 234498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7517.159107] LustreError: 234497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7520.749332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7521.087471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7525.216181] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7527.853622] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7534.350759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7534.665792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7540.235894] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7547.754528] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7547.919379] Lustre: lustre-OST0001: new disk, initializing [ 7547.929870] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7547.936556] Lustre: 236803:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7552.837478] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7556.190554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7556.202554] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7556.286414] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7563.159080] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7567.348723] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7578.614257] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 16:04:42 (1787083482) [ 7582.190770] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7582.198109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7582.204603] Lustre: Skipped 3 previous similar messages [ 7582.215246] Lustre: Skipped 3 previous similar messages [ 7586.412465] Lustre: server umount lustre-MDT0000 complete [ 7587.302179] LustreError: 234498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7587.318852] LustreError: 234498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7589.339226] LustreError: 234479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787083494 with bad export cookie 2209941076095281851 [ 7589.343627] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7589.355809] LustreError: 234479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7589.628923] Lustre: server umount lustre-MDT0001 complete [ 7603.092539] Lustre: server umount lustre-OST0000 complete [ 7616.956526] Lustre: server umount lustre-OST0001 complete [ 7623.085761] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 7632.062787] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 7661.753903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7661.917115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7661.946952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7662.010917] Lustre: lustre-MDT0000: new disk, initializing [ 7662.061027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7662.065618] Lustre: Skipped 1 previous similar message [ 7662.087259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7665.594124] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7674.254336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7674.400318] Lustre: 240750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7674.456457] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7674.460760] Lustre: Skipped 1 previous similar message [ 7674.593220] Lustre: lustre-MDT0001: new disk, initializing [ 7674.705814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7674.717658] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7678.577855] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7686.615801] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7692.358046] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7698.870706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7699.102650] Lustre: lustre-OST0000: new disk, initializing [ 7699.107944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7699.114108] Lustre: 242720:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7699.190233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7699.196631] Lustre: Skipped 1 previous similar message [ 7700.424946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7700.434357] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7700.524376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7704.784414] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7713.220487] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7717.427298] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7717.671946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7721.899284] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7722.084876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7726.049838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7726.058242] LustreError: Skipped 1 previous similar message [ 7726.068766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7726.076043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7726.093722] Lustre: Skipped 1 previous similar message [ 7726.104246] Lustre: Skipped 1 previous similar message [ 7729.895322] Lustre: server umount lustre-OST0000 complete [ 7736.298263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7736.302481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7736.317378] Lustre: Skipped 1 previous similar message [ 7736.338377] LustreError: Skipped 1 previous similar message [ 7739.655213] Lustre: server umount lustre-MDT0000 complete [ 7741.412218] LustreError: 240757:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7741.429825] LustreError: 240757:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7743.158884] LustreError: 240743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787083648 with bad export cookie 2209941076095283986 [ 7743.160817] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7743.168395] LustreError: 240743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7743.496228] Lustre: server umount lustre-MDT0001 complete [ 7750.441443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7750.759993] LustreError: 244936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7754.401506] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7761.444927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7765.208884] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7770.984468] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7775.682179] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7781.181826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7781.471290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7781.474895] Lustre: Skipped 2 previous similar messages [ 7786.573983] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 7792.219746] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7939.556038] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7939.562182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7939.565356] Lustre: Skipped 1 previous similar message [ 7939.566152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7939.579016] Lustre: Skipped 3 previous similar messages [ 7951.329155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7951.436564] Lustre: server umount lustre-OST0000 complete [ 7968.224163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7968.345666] Lustre: server umount lustre-MDT0000 complete [ 7970.274559] LustreError: 245664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7970.292246] LustreError: 245664:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7970.956742] LustreError: 244917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787083875 with bad export cookie 2209941076095284868 [ 7970.962394] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7970.967384] LustreError: 244917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7971.229827] Lustre: server umount lustre-MDT0001 complete [ 8058.785056] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 8060.378221] Key type lgssc unregistered [ 8060.587897] LNet: 248547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8060.592757] LNetError: 248547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8060.603693] LNet: Removed LNI 192.168.204.149@tcp [ 8061.066168] Key type .llcrypt unregistered [ 8061.069199] Key type ._llcrypt unregistered [ 8077.371151] Key type ._llcrypt registered [ 8077.374556] Key type .llcrypt registered [ 8077.464977] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 8087.988761] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 8088.489381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8088.538798] alg: No test for adler32 (adler32-zlib) [ 8089.449128] Lustre: Lustre: Build Version: 2.17.57_1_g711d631 [ 8089.575965] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 8091.192233] Key type lgssc registered [ 8091.889823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8119.906327] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 8125.626676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8125.642537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8126.778105] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8126.799963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8126.848623] Lustre: lustre-MDT0000: new disk, initializing [ 8126.894562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8126.906225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8129.366418] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8136.198217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8136.273284] Lustre: 252589:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8136.318429] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8136.326043] Lustre: Skipped 1 previous similar message [ 8136.395079] Lustre: lustre-MDT0001: new disk, initializing [ 8136.457930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8136.484070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8136.489604] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8139.200412] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8142.778884] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8147.634439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8147.836799] Lustre: lustre-OST0000: new disk, initializing [ 8147.841657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8147.847038] Lustre: 254221:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8147.914783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8150.378493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8150.384785] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8150.469063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8151.442473] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8158.396398] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8158.492408] Lustre: lustre-OST0001: new disk, initializing [ 8158.495826] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8158.501262] Lustre: 255092:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8158.569149] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8162.356236] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8164.387141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8164.394267] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8164.449088] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8170.604756] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8174.490471] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8179.685755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8179.688067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8179.692408] Lustre: Skipped 3 previous similar messages [ 8179.698790] Lustre: Skipped 3 previous similar messages [ 8184.802458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8184.809105] Lustre: Skipped 3 previous similar messages [ 8185.850939] Lustre: server umount lustre-MDT0000 complete [ 8187.653693] LustreError: 252582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787084092 with bad export cookie 3174736012805577126 [ 8187.655795] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8187.662320] LustreError: 252582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8187.823104] Lustre: server umount lustre-MDT0001 complete [ 8199.135928] Lustre: server umount lustre-OST0000 complete [ 8210.193266] Lustre: server umount lustre-OST0001 complete [ 8217.751963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8223.035628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8227.131507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8232.461865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8243.700823] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 8249.643302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8249.659204] Lustre: lustre-MDT0000: reset Object Index mappings [ 8249.954559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8252.796979] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8254.950410] LustreError: 260099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8254.962443] LustreError: 260099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8257.796197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8257.810535] Lustre: lustre-MDT0001: reset Object Index mappings [ 8258.035008] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8260.497870] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8262.181580] Lustre: 261243:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8265.824714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8267.048816] LustreError: 261594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8269.223809] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8272.357139] LustreError: 261597:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8273.584447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8274.797882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8274.802036] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8276.600818] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8281.255777] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8283.021874] Lustre: 263111:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8285.161181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8285.164243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8285.171611] Lustre: Skipped 3 previous similar messages [ 8285.177825] Lustre: Skipped 2 previous similar messages [ 8290.274177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8290.277901] Lustre: Skipped 2 previous similar messages [ 8290.893614] Lustre: server umount lustre-MDT0000 complete [ 8292.821018] LustreError: 260081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787084197 with bad export cookie 3174736012805583916 [ 8292.825677] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8292.828586] LustreError: 260081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8293.006735] Lustre: server umount lustre-MDT0001 complete [ 8305.700627] Lustre: server umount lustre-OST0000 complete [ 8317.125439] Lustre: server umount lustre-OST0001 complete [ 8326.537740] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 16:17:11 (1787084231) [ 8333.609250] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 8338.844689] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 8357.140556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8357.249142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8357.269236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8357.312537] Lustre: lustre-MDT0000: new disk, initializing [ 8357.354963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8357.359738] Lustre: Skipped 2 previous similar messages [ 8357.371495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8359.171421] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8364.710710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8364.786764] Lustre: 267358:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8364.932187] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8366.709458] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8371.408139] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8373.925501] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8376.954626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8377.108549] Lustre: lustre-OST0000: new disk, initializing [ 8377.110341] Lustre: Skipped 1 previous similar message [ 8377.113288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8377.116323] Lustre: Skipped 2 previous similar messages [ 8377.119570] Lustre: 269327:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8378.309283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8378.313882] Lustre: Skipped 1 previous similar message [ 8378.317599] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8378.374606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8379.759328] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8384.392234] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8386.407855] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8386.526736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8388.411292] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8388.518805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8393.698209] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8393.699251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8393.701936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8393.701945] Lustre: Skipped 1 previous similar message [ 8393.702546] LustreError: Skipped 1 previous similar message [ 8393.710436] Lustre: Skipped 1 previous similar message [ 8395.513197] Lustre: server umount lustre-OST0000 complete [ 8398.818375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8398.825255] Lustre: Skipped 1 previous similar message [ 8403.577592] Lustre: server umount lustre-MDT0000 complete [ 8403.938036] LustreError: 267369:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8403.946144] LustreError: 267369:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8405.432456] LustreError: 267348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787084310 with bad export cookie 3174736012805585981 [ 8405.436842] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8405.441201] LustreError: 267348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8405.605935] Lustre: server umount lustre-MDT0001 complete [ 8415.045752] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 8420.441460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8420.669229] LustreError: 272255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8420.696090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8420.699174] Lustre: Skipped 2 previous similar messages [ 8422.738601] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8427.213849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8429.426415] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8430.773643] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8433.860308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8436.892315] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8441.095448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8441.185565] Lustre: lustre-OST0001: new disk, initializing [ 8441.188222] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8441.192784] Lustre: 274560:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8442.348337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8442.354225] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8442.409255] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8444.292834] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8450.208669] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8452.085230] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8454.730046] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8454.746219] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8458.209614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8458.214301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8458.222268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8458.225426] Lustre: Skipped 3 previous similar messages [ 8463.668049] Lustre: server umount lustre-MDT0000 complete [ 8465.465602] LustreError: 274559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787084370 with bad export cookie 3174736012805586870 [ 8465.471696] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8465.627521] Lustre: server umount lustre-MDT0001 complete [ 8477.590824] Lustre: server umount lustre-OST0000 complete [ 8489.796504] Lustre: server umount lustre-OST0001 complete [ 8497.696796] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 8502.756838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8502.938103] LustreError: 277609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8502.946372] LustreError: 277609:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8502.979710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8502.982464] Lustre: Skipped 3 previous similar messages [ 8504.982335] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8509.230540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8511.388025] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8512.874550] Lustre: 278751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8516.242219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8519.038377] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8519.524594] LustreError: 279105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8519.540652] LustreError: 279105:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8523.105449] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8526.158166] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8527.347948] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 8527.354955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 8535.439078] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8537.197556] Lustre: 280626:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8539.845161] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 8539.854591] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 8544.430156] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 16:20:49 (1787084449) [ 8558.051627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8558.054063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8558.057994] Lustre: Skipped 5 previous similar messages [ 8558.063416] Lustre: Skipped 5 previous similar messages [ 8562.600360] Lustre: server umount lustre-MDT0000 complete [ 8563.171185] LustreError: 277628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8563.178589] LustreError: 277628:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8564.147821] LustreError: 278754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787084469 with bad export cookie 3174736012805593219 [ 8564.149782] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8564.152721] LustreError: 278754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8570.373170] Lustre: server umount lustre-MDT0001 complete [ 8578.244742] Lustre: server umount lustre-OST0000 complete [ 8593.888302] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8593.975077] Lustre: server umount lustre-OST0001 complete [ 8596.959459] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 8601.652537] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 8620.278979] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 8624.336099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8624.428026] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8624.442156] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8624.480331] Lustre: lustre-MDT0000: new disk, initializing [ 8624.524209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8626.050545] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8630.918435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8630.969848] Lustre: 284865:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8631.085141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8631.089044] Lustre: Skipped 4 previous similar messages [ 8631.106883] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8632.786235] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8638.599953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8638.734793] Lustre: 286499:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8638.895991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8641.445320] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8646.967128] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8647.065586] Lustre: 287370:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8648.547207] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8650.228865] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8655.598596] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8657.191702] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8657.194740] Lustre: Skipped 1 previous similar message [ 8662.513813] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 16:22:47 (1787084567) [ 8664.035877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8664.038769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8664.046436] Lustre: Skipped 6 previous similar messages [ 8664.049539] Lustre: Skipped 6 previous similar messages [ 8669.647327] Lustre: server umount lustre-MDT0000 complete [ 8671.339296] LustreError: 285661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787084576 with bad export cookie 3174736012805601101 [ 8671.343433] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8671.344819] LustreError: 285661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8671.472525] Lustre: server umount lustre-MDT0001 complete [ 8683.532692] Lustre: server umount lustre-OST0000 complete [ 8695.403804] Lustre: server umount lustre-OST0001 complete [ 8698.845513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8699.021624] LustreError: 289614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8699.030579] LustreError: 289614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8700.657241] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8704.135946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8705.941018] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8708.851121] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8711.028477] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8713.744338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8716.252890] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8718.959846] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8736.483875] loop1: detected capacity change from 0 to 204800000 [ 8736.575948] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8736.583582] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8736.590100] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8738.722115] loop1: detected capacity change from 0 to 204800000 [ 8741.111143] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8741.149783] LustreError: 289604:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs14831: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 8741.157596] LustreError: 289604:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs14831 [ 8741.161860] LustreError: 289604:0:(mgs_handler.c:517:mgs_target_reg()) fs14831-MDT0001: Error checking target index: rc = -13 [ 8741.167092] LustreError: fs14831-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 8741.172706] LustreError: 293354:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 8741.177422] LustreError: 293354:0:(tgt_mount.c:2129:server_put_super()) no obd fs14831-MDT0001 [ 8741.181219] LustreError: 293354:0:(tgt_mount.c:116:server_deregister_mount()) fs14831-MDT0001 not registered [ 8741.198038] Lustre: server umount fs14831-MDT0001 complete [ 8741.200748] LustreError: 293354:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 8744.800705] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8744.847616] Lustre: Setting parameter fs14831-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs14831-MDT0001 [ 8744.876632] Lustre: srv-fs14831-MDT0001: No data found on store. Initialize space. [ 8744.879407] Lustre: Skipped 4 previous similar messages [ 8744.918564] Lustre: fs14831-MDT0001: new disk, initializing [ 8744.920658] Lustre: Skipped 3 previous similar messages [ 8744.980209] Lustre: srv-fs14831-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8744.983354] Lustre: 293819:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14831-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8744.987296] LustreError: 293819:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14831-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8746.642862] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8749.920482] Lustre: server umount fs14831-MDT0001 complete [ 8755.377595] loop1: detected capacity change from 0 to 204800000 [ 8757.849024] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8758.002927] Lustre: 295384:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14831-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8758.008529] LustreError: 295384:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14831-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8759.726472] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8761.864626] Lustre: server umount fs14831-MDT0001 complete [ 8765.921962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8765.925960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8765.932600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8765.935410] Lustre: Skipped 4 previous similar messages [ 8780.256149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8780.345723] Lustre: server umount lustre-MDT0000 complete [ 8784.288873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8784.361092] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8786.427960] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8789.996936] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8790.339899] loop1: detected capacity change from 0 to 204800000 [ 8792.845280] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8793.002854] Lustre: 297825:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14831-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8793.008747] LustreError: 297825:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14831-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8794.868439] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8796.969649] Lustre: server umount fs14831-MDT0001 complete [ 8805.348970] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 8809.958084] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8809.964822] LustreError: Skipped 1 previous similar message [ 8812.193929] Lustre: server umount lustre-OST0000 complete [ 8819.799516] Lustre: server umount lustre-MDT0000 complete [ 8821.035310] LustreError: 289594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787084726 with bad export cookie 3174736012805603782 [ 8821.039232] LustreError: 289594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8821.039557] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8821.164682] Lustre: server umount lustre-MDT0001 complete [ 8825.218961] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 8826.305479] Key type lgssc unregistered [ 8826.438400] LNet: 300169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8826.442066] LNetError: 300169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8826.448285] LNet: Removed LNI 192.168.204.149@tcp [ 8826.794143] Key type .llcrypt unregistered [ 8826.795493] Key type ._llcrypt unregistered [ 8835.503859] Key type ._llcrypt registered [ 8835.505361] Key type .llcrypt registered [ 8835.552124] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 8835.892229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8835.930036] alg: No test for adler32 (adler32-zlib) [ 8836.803633] Lustre: Lustre: Build Version: 2.17.57_1_g711d631 [ 8836.905876] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 8838.489057] Key type lgssc registered [ 8838.954948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8842.118631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8842.126144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8843.325557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8845.011257] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8848.724888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8848.936439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8850.736666] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8854.142896] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8856.600554] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8859.417168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8859.566893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8862.035782] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8865.088324] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8868.771501] LustreError: 303244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8868.782395] LustreError: 303244:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8871.773344] LustreError: 303244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8873.890723] LustreError: 303245:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8873.900468] LustreError: 303245:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8876.907316] LustreError: 303246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8882.027422] LustreError: 303246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8882.034647] LustreError: 303246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8882.451062] loop1: detected capacity change from 0 to 204800000 [ 8882.525684] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8882.530739] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8882.536088] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8884.682733] loop1: detected capacity change from 0 to 204800000 [ 8887.110762] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8887.188708] Lustre: fs7107-OST0000: new disk, initializing [ 8887.191300] Lustre: srv-fs7107-OST0000: No data found on store. Initialize space. [ 8887.195135] Lustre: 305172:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8887.225839] Lustre: fs7107-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8889.728835] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8892.267869] LustreError: 303920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8892.276767] LustreError: 303920:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8893.979790] Lustre: server umount fs7107-OST0000 complete [ 8899.733750] loop1: detected capacity change from 0 to 204800000 [ 8902.218507] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8902.306749] Lustre: fs7107-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8904.796221] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8909.009044] loop2: detected capacity change from 0 to 204800000 [ 8909.107945] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8909.113599] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8909.121807] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8909.796073] LustreError: 303246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8909.808838] LustreError: 303246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 8911.533947] loop2: detected capacity change from 0 to 204800000 [ 8914.053023] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8914.088466] LustreError: fs7107-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 8914.094564] LustreError: fs7107-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 8914.099453] LustreError: 308208:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 8914.103583] LustreError: 308208:0:(tgt_mount.c:2129:server_put_super()) no obd fs7107-OST0001 [ 8914.106783] LustreError: 308208:0:(tgt_mount.c:116:server_deregister_mount()) fs7107-OST0001 not registered [ 8914.120970] Lustre: server umount fs7107-OST0001 complete [ 8914.122640] LustreError: 308208:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 8918.424073] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8918.491917] Lustre: fs7107-OST0001: new disk, initializing [ 8918.494865] Lustre: srv-fs7107-OST0001: No data found on store. Initialize space. [ 8918.500483] Lustre: 308760:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8918.535458] Lustre: fs7107-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8920.902548] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8925.243888] Lustre: server umount fs7107-OST0000 complete [ 8929.326524] Lustre: server umount fs7107-OST0001 complete [ 8934.786898] loop1: detected capacity change from 0 to 204800000 [ 8937.255050] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8937.348919] Lustre: fs7107-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8939.904895] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8943.192762] loop2: detected capacity change from 0 to 204800000 [ 8943.468490] LustreError: 303920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8943.482867] LustreError: 303920:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 8946.156812] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8949.343243] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 8954.200520] Lustre: server umount fs7107-OST0000 complete [ 8955.885808] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 8958.687084] Lustre: server umount fs7107-OST0001 complete [ 8960.999691] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 8966.114765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8966.116427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8966.129826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8966.134700] Lustre: Skipped 1 previous similar message [ 8970.730139] Lustre: server umount lustre-OST0000 complete [ 8976.355499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8976.357691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8976.366158] Lustre: Skipped 2 previous similar messages [ 8976.372460] Lustre: Skipped 1 previous similar message [ 8980.852434] Lustre: server umount lustre-MDT0000 complete [ 8982.491208] LustreError: 301398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787084887 with bad export cookie 7527153295010957816 [ 8982.493529] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8982.502380] LustreError: 301398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8982.640660] Lustre: server umount lustre-MDT0001 complete [ 8987.343315] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 8988.586106] Key type lgssc unregistered [ 8988.742642] LNet: 314674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8988.746749] LNetError: 314674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8988.762504] LNet: Removed LNI 192.168.204.149@tcp [ 8989.122340] Key type .llcrypt unregistered [ 8989.124255] Key type ._llcrypt unregistered [ 9002.147148] Key type ._llcrypt registered [ 9002.148533] Key type .llcrypt registered [ 9002.200792] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 9009.749066] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 9010.194502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9010.238842] alg: No test for adler32 (adler32-zlib) [ 9011.129345] Lustre: Lustre: Build Version: 2.17.57_1_g711d631 [ 9011.244351] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 9012.832422] Key type lgssc registered [ 9013.304309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9029.284480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9029.297470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9030.405782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9030.425843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9030.466070] Lustre: lustre-MDT0000: new disk, initializing [ 9030.512403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9030.521600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9032.573977] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9038.542057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9038.629361] Lustre: 318506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9038.665351] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9038.668234] Lustre: Skipped 1 previous similar message [ 9038.721827] Lustre: lustre-MDT0001: new disk, initializing [ 9038.780851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9038.801612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9038.808061] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9040.817081] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9046.010431] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9048.590282] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9051.765606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9051.946534] Lustre: lustre-OST0000: new disk, initializing [ 9051.948677] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9051.952272] Lustre: 320475:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9052.002272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9054.927751] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9059.720921] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9060.781797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9060.788893] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9060.844085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9065.980526] Lustre: Mounted lustre-client - version 2.17.57_1_g711d631 [ 9066.455352] Lustre: Unmounted lustre-client [ 9071.075166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9071.077406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9071.081469] LustreError: Skipped 1 previous similar message [ 9071.082298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9071.088192] Lustre: Skipped 1 previous similar message [ 9071.095853] Lustre: Skipped 1 previous similar message [ 9072.878760] Lustre: server umount lustre-OST0000 complete [ 9076.193960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9076.195917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9076.202035] Lustre: Skipped 1 previous similar message [ 9081.032217] Lustre: server umount lustre-MDT0000 complete [ 9082.890355] LustreError: 319300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787084987 with bad export cookie 307411792081892985 [ 9082.892479] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9082.898758] LustreError: 319300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9083.043192] Lustre: server umount lustre-MDT0001 complete [ 9088.179860] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 9089.566741] Key type lgssc unregistered [ 9116.704452] LNet: 1 local NIs in recovery (showing 1): 192.168.204.149@tcp [ 9117.409313] LNet: 322635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9117.413431] LNetError: 322635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9193.190373] LNet: Removed LNI 192.168.204.149@tcp [ 9193.661531] Key type .llcrypt unregistered [ 9193.662830] Key type ._llcrypt unregistered [ 9201.745955] Key type ._llcrypt registered [ 9201.747549] Key type .llcrypt registered [ 9201.812907] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 9209.769822] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 9210.206048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9210.262234] alg: No test for adler32 (adler32-zlib) [ 9211.153288] Lustre: Lustre: Build Version: 2.17.57_1_g711d631 [ 9211.260749] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 9212.856434] Key type lgssc registered [ 9213.410441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9226.976931] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 16:32:11 (1787085131) [ 9229.887431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9229.901682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9231.012470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9231.033429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9231.079323] Lustre: lustre-MDT0000: new disk, initializing [ 9231.112350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9231.120738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9233.067310] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9238.595242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9238.652101] Lustre: 325831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9238.683838] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9238.687561] Lustre: Skipped 1 previous similar message [ 9238.733992] Lustre: lustre-MDT0001: new disk, initializing [ 9238.790734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9238.810321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9238.814971] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9240.690067] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9245.339730] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9247.651175] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9251.001204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9251.133915] Lustre: lustre-OST0000: new disk, initializing [ 9251.136821] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9251.141987] Lustre: 327800:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9251.187780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9254.005983] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9258.874226] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9260.043491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9260.049296] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9260.120681] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9275.365127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9275.366332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9275.368386] LustreError: Skipped 1 previous similar message [ 9275.375126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9275.378927] Lustre: Skipped 1 previous similar message [ 9276.840403] Lustre: server umount lustre-OST0000 complete [ 9279.968975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9279.972344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9279.978689] Lustre: Skipped 1 previous similar message [ 9279.981357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9284.842223] Lustre: server umount lustre-MDT0000 complete [ 9286.456547] LustreError: 325824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787085191 with bad export cookie 2722316929174353499 [ 9286.458664] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9286.464291] LustreError: 325824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9286.603204] Lustre: server umount lustre-MDT0001 complete [ 9291.311088] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 9292.551135] Key type lgssc unregistered [ 9292.679580] LNet: 329710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9292.683864] LNetError: 329710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9292.693310] LNet: Removed LNI 192.168.204.149@tcp [ 9293.059375] Key type .llcrypt unregistered [ 9293.060908] Key type ._llcrypt unregistered [ 9303.560565] Key type ._llcrypt registered [ 9303.562489] Key type .llcrypt registered [ 9303.618756] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 9304.012142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9304.040819] alg: No test for adler32 (adler32-zlib) [ 9304.933988] Lustre: Lustre: Build Version: 2.17.57_1_g711d631 [ 9305.047311] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 9306.648400] Key type lgssc registered [ 9307.151971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9310.672767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9310.681475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9311.905996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9313.671767] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9317.275670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9317.468984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9319.314570] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9322.398294] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9324.801468] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9327.668502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9327.712815] LustreError: 332777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9327.723906] LustreError: 332777:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9327.825626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9330.412966] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9333.606053] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9343.911618] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9344.623724] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9346.531312] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9346.532991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9346.537798] Lustre: Skipped 2 previous similar messages [ 9346.540991] Lustre: Skipped 1 previous similar message [ 9350.523818] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.49@tcp (stopping) [ 9350.528959] Lustre: Skipped 1 previous similar message [ 9351.649840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9351.653168] Lustre: Skipped 2 previous similar messages [ 9352.420070] Lustre: server umount lustre-MDT0000 complete [ 9354.093570] LustreError: 330943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787085259 with bad export cookie 14103317210217843810 [ 9354.096387] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9354.108179] LustreError: 330943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9355.626381] LustreError: 330956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9355.634906] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.49@tcp (stopping) [ 9356.769175] LustreError: 332200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9356.769749] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9359.777318] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9359.777436] LustreError: 330956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9359.780166] Lustre: Skipped 1 previous similar message [ 9360.332326] Lustre: server umount lustre-MDT0001 complete [ 9376.224131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9376.293638] Lustre: server umount lustre-OST0000 complete [ 9380.398577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9382.307634] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9383.512593] Lustre: 335137:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.149@tcp [ 9384.504815] Lustre: server umount MGS complete [ 9388.101618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9388.277603] LustreError: 335708:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9388.304199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9389.997939] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9393.616325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9393.780353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9395.553349] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9399.075176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9401.526187] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9404.652342] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9405.343387] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 9407.633190] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9417.538280] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 9423.844355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9423.845795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9423.846810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9423.853598] Lustre: Skipped 1 previous similar message [ 9423.858589] LustreError: Skipped 1 previous similar message [ 9424.979893] Lustre: server umount lustre-OST0000 complete [ 9428.961637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9428.968466] Lustre: Skipped 1 previous similar message [ 9432.540705] Lustre: server umount lustre-MDT0000 complete [ 9434.027316] LustreError: 335688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787085339 with bad export cookie 14103317210217844636 [ 9434.031232] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9434.033587] LustreError: 335688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9434.166246] Lustre: server umount lustre-MDT0001 complete [ 9438.697585] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 9439.875883] Key type lgssc unregistered [ 9440.008886] LNet: 339144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9440.011870] LNetError: 339144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9440.021690] LNet: Removed LNI 192.168.204.149@tcp [ 9440.373356] Key type .llcrypt unregistered [ 9440.374790] Key type ._llcrypt unregistered [ 9447.250620] Key type ._llcrypt registered [ 9447.252254] Key type .llcrypt registered [ 9447.292748] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 9454.797514] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 9455.183553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9455.229159] alg: No test for adler32 (adler32-zlib) [ 9456.092094] Lustre: Lustre: Build Version: 2.17.57_1_g711d631 [ 9456.183547] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 9457.768183] Key type lgssc registered [ 9458.198583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9472.929455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9472.939357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9474.032318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9474.049853] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9474.091051] Lustre: lustre-MDT0000: new disk, initializing [ 9474.126589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9474.133966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9475.736475] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9480.484359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9480.529956] Lustre: 342278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9480.564782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9480.566821] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9480.567263] Lustre: Skipped 1 previous similar message [ 9480.608061] Lustre: lustre-MDT0001: new disk, initializing [ 9480.643584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9480.657826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9480.661385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9482.177678] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9486.395844] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9488.500976] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9491.268070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9491.398599] Lustre: lustre-OST0000: new disk, initializing [ 9491.401432] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9491.405505] Lustre: 344245:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9491.441703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9493.756772] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9496.067693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9496.072322] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9496.110409] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9497.830597] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9499.523495] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9499.606764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9501.271941] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9501.356724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9506.275631] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9506.276159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9506.276966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9506.279643] Lustre: Skipped 1 previous similar message [ 9506.281966] LustreError: Skipped 1 previous similar message [ 9508.182996] Lustre: server umount lustre-OST0000 complete [ 9511.394914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9511.395253] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9511.396307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9511.396313] Lustre: Skipped 1 previous similar message [ 9511.400762] Lustre: Skipped 1 previous similar message [ 9515.678231] Lustre: server umount lustre-MDT0000 complete [ 9517.158789] LustreError: 342270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787085422 with bad export cookie 9221856422400251582 [ 9517.160423] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9517.164079] LustreError: 342270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9517.294841] Lustre: server umount lustre-MDT0001 complete [ 9520.866151] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 16:37:05 (1787085425) [ 9527.295524] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 9531.263115] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 9544.056241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9544.142938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9544.153093] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9544.185988] Lustre: lustre-MDT0000: new disk, initializing [ 9544.208077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9544.213828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9545.609548] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9549.814189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9549.855909] Lustre: 348998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9549.882235] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9549.885143] Lustre: Skipped 1 previous similar message [ 9549.923243] Lustre: lustre-MDT0001: new disk, initializing [ 9549.956463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9549.970767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9549.975911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9551.389511] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9555.226793] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9557.269035] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9559.755356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9559.854657] Lustre: lustre-OST0000: new disk, initializing [ 9559.857497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9559.862174] Lustre: 350966:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9559.893486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9562.064278] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9566.241438] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9568.250621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9568.256067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9568.299590] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9583.588351] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9583.590085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9583.591863] LustreError: Skipped 1 previous similar message [ 9583.598480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9583.603165] Lustre: Skipped 2 previous similar messages [ 9588.316494] Lustre: server umount lustre-OST0000 complete [ 9591.264788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9591.267399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9591.271404] Lustre: Skipped 1 previous similar message [ 9591.273407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9595.869147] Lustre: server umount lustre-MDT0000 complete [ 9597.203097] LustreError: 348992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787085502 with bad export cookie 9221856422400252464 [ 9597.206113] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9597.207089] LustreError: 348992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9597.326208] Lustre: server umount lustre-MDT0001 complete [ 9601.740828] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 9602.892507] Key type lgssc unregistered [ 9603.033390] LNet: 353117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9603.037181] LNetError: 353117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9603.050348] LNet: Removed LNI 192.168.204.149@tcp [ 9603.401334] Key type .llcrypt unregistered [ 9603.402559] Key type ._llcrypt unregistered [ 9641.425318] Key type ._llcrypt registered [ 9641.426947] Key type .llcrypt registered [ 9641.470688] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 9641.605351] Key type .llcrypt unregistered [ 9641.607197] Key type ._llcrypt unregistered [ 9673.882681] Key type ._llcrypt registered [ 9673.884260] Key type .llcrypt registered [ 9673.940848] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 9674.121596] Key type .llcrypt unregistered [ 9674.124246] Key type ._llcrypt unregistered [ 9688.046203] Key type ._llcrypt registered [ 9688.047257] Key type .llcrypt registered [ 9688.092330] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 9688.220536] Key type .llcrypt unregistered [ 9688.223102] Key type ._llcrypt unregistered [ 9702.159706] Key type ._llcrypt registered [ 9702.161083] Key type .llcrypt registered [ 9702.220156] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 9702.370691] Key type .llcrypt unregistered [ 9702.372689] Key type ._llcrypt unregistered [ 9734.897175] Key type ._llcrypt registered [ 9734.898526] Key type .llcrypt registered [ 9734.936465] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 9735.051401] Key type .llcrypt unregistered [ 9735.053395] Key type ._llcrypt unregistered [ 9747.922594] Key type ._llcrypt registered [ 9747.924244] Key type .llcrypt registered [ 9747.976146] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 9748.127897] Key type .llcrypt unregistered [ 9748.129926] Key type ._llcrypt unregistered [ 9760.080300] Key type ._llcrypt registered [ 9760.081545] Key type .llcrypt registered [ 9760.122869] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 9760.253319] Key type .llcrypt unregistered [ 9760.255495] Key type ._llcrypt unregistered [ 9772.404604] Key type ._llcrypt registered [ 9772.406523] Key type .llcrypt registered [ 9772.459464] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 9772.588457] Key type .llcrypt unregistered [ 9772.589944] Key type ._llcrypt unregistered [ 9784.965516] Key type ._llcrypt registered [ 9784.966884] Key type .llcrypt registered [ 9785.006243] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 9785.144683] Key type .llcrypt unregistered [ 9785.146968] Key type ._llcrypt unregistered [ 9797.093275] Key type ._llcrypt registered [ 9797.094397] Key type .llcrypt registered [ 9797.132182] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 9797.248550] Key type .llcrypt unregistered [ 9797.250191] Key type ._llcrypt unregistered [ 9820.358220] Key type ._llcrypt registered [ 9820.359808] Key type .llcrypt registered [ 9820.405338] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 9820.537710] Key type .llcrypt unregistered [ 9820.539147] Key type ._llcrypt unregistered [ 9832.638664] Key type ._llcrypt registered [ 9832.640383] Key type .llcrypt registered [ 9832.681093] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 9832.794411] Key type .llcrypt unregistered [ 9832.795347] Key type ._llcrypt unregistered [ 9844.390201] Key type ._llcrypt registered [ 9844.392081] Key type .llcrypt registered [ 9844.433493] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 9844.568506] Key type .llcrypt unregistered [ 9844.570057] Key type ._llcrypt unregistered [ 9865.662183] Key type ._llcrypt registered [ 9865.663611] Key type .llcrypt registered [ 9865.715942] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 9872.843723] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 9873.267701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9873.290089] alg: No test for adler32 (adler32-zlib) [ 9874.153726] Lustre: Lustre: Build Version: 2.17.57_1_g711d631 [ 9874.245147] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 9875.832166] Key type lgssc registered [ 9876.227826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9888.509253] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9888.515574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9889.610797] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 9889.621443] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 9889.658341] Lustre: tst250fs-MDT0000: new disk, initializing [ 9889.679110] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9889.686125] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9891.044838] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9895.127205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9895.165579] Lustre: 373074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9895.189747] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 9895.192696] Lustre: Skipped 1 previous similar message [ 9895.231882] Lustre: tst250fs-MDT0001: new disk, initializing [ 9895.263453] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9895.274982] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9895.278168] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9896.573829] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9898.820358] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9901.159174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9901.269882] Lustre: tst250fs-OST0000: new disk, initializing [ 9901.272377] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 9901.276699] Lustre: 374705:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9901.316178] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9903.275555] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9904.183681] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9904.186785] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9904.222825] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9907.356684] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9907.425843] Lustre: tst250fs-OST0001: new disk, initializing [ 9907.427510] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 9907.430849] Lustre: 375579:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9907.465971] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9909.422841] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug -1 all [ 9912.005684] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 9913.428211] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9913.431731] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9913.493495] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9924.070412] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9924.073074] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 9924.074651] Lustre: Skipped 3 previous similar messages [ 9924.076694] Lustre: Skipped 2 previous similar messages [ 9928.728347] Lustre: server umount tst250fs-MDT0000 complete [ 9930.013532] LustreError: 375577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787085835 with bad export cookie 5591233951493047339 [ 9930.017056] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9930.019585] LustreError: 375577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9930.134335] Lustre: server umount tst250fs-MDT0001 complete [ 9941.514191] Lustre: server umount tst250fs-OST0000 complete [ 9952.098111] Lustre: server umount tst250fs-OST0001 complete [ 9958.727206] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 9961.778480] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 9972.739774] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 16:44:37 (1787085877) [ 9973.315442] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 9979.437222] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 9983.716750] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 9995.287549] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9736 sec ========== 16:44:59 (1787085899) [ 9995.941235] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:45:00 (1787085900) === [ 9997.265563] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:45:01 (1787085901) === [10010.486613] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [10011.463920] Key type lgssc unregistered [10011.595304] LNet: 384712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10011.597622] LNetError: 384712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10011.607356] LNet: Removed LNI 192.168.204.149@tcp [10011.919120] Key type .llcrypt unregistered [10011.921166] Key type ._llcrypt unregistered