[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 673084403 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002388] x2apic enabled [ 0.003012] Switched APIC routing to physical x2apic. [ 0.004020] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009021] pid_max: default: 32768 minimum: 301 [ 0.010157] LSM: Security Framework initializing [ 0.011093] Yama: becoming mindful. [ 0.012084] SELinux: Initializing. [ 0.014049] *** VALIDATE selinux *** [ 0.027140] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.035074] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.036175] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.037132] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.038150] *** VALIDATE tmpfs *** [ 0.041031] *** VALIDATE proc *** [ 0.044029] *** VALIDATE cgroup *** [ 0.045023] *** VALIDATE cgroup2 *** [ 0.046482] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.047187] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.048082] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.049046] Spectre V2 : User space: Vulnerable [ 0.051012] Speculative Store Bypass: Vulnerable [ 0.054553] debug: unmapping init [mem 0xffffffff8bc59000-0xffffffff8bc60fff] [ 0.057000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.057886] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.059034] ... version: 2 [ 0.060019] ... bit width: 48 [ 0.061096] ... generic registers: 4 [ 0.062020] ... value mask: 0000ffffffffffff [ 0.063023] ... max period: 00007fffffffffff [ 0.064026] ... fixed-purpose events: 3 [ 0.065019] ... event mask: 000000070000000f [ 0.066432] rcu: Hierarchical SRCU implementation. [ 0.070536] smp: Bringing up secondary CPUs ... [ 0.071751] x86: Booting SMP configuration: [ 0.072045] .... node #0, CPUs: #1 #2 #3 [ 0.087107] smp: Brought up 1 node, 4 CPUs [ 0.089022] smpboot: Max logical packages: 1 [ 0.090024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.162135] node 0 deferred pages initialised in 69ms [ 0.168088] devtmpfs: initialized [ 0.172279] x86/mm: Memory block size: 128MB [ 0.177255] gcov: version magic: 0x41383552 [ 0.180385] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.181103] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.182614] pinctrl core: initialized pinctrl subsystem [ 0.183811] [ 0.184027] ************************************************************* [ 0.185101] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.186023] ** ** [ 0.187020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.188094] ** ** [ 0.189026] ** This means that this kernel is built to expose internal ** [ 0.190028] ** IOMMU data structures, which may compromise security on ** [ 0.191027] ** your system. ** [ 0.192024] ** ** [ 0.193020] ** If you see this message and you are not debugging the ** [ 0.194025] ** kernel, report this immediately to your vendor! ** [ 0.195018] ** ** [ 0.196021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.197020] ************************************************************* [ 0.199459] NET: Registered protocol family 16 [ 0.200758] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.201176] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.202167] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.203619] cpuidle: using governor menu [ 0.207000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.212897] PCI: Using configuration type 1 for base access [ 0.217338] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.233123] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.236091] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.250168] cryptd: max_cpu_qlen set to 1000 [ 0.253458] ACPI: Added _OSI(Module Device) [ 0.259027] ACPI: Added _OSI(Processor Device) [ 0.263022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.265019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.270339] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.278871] ACPI: Interpreter enabled [ 0.279094] ACPI: PM: (supports S0 S3 S4 S5) [ 0.280026] ACPI: Using IOAPIC for interrupt routing [ 0.281120] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.282484] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.296016] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.298248] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.302025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.306099] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.312711] acpiphp: Slot [2] registered [ 0.315275] acpiphp: Slot [5] registered [ 0.317254] acpiphp: Slot [6] registered [ 0.319243] acpiphp: Slot [7] registered [ 0.321194] acpiphp: Slot [8] registered [ 0.322396] acpiphp: Slot [9] registered [ 0.324326] acpiphp: Slot [10] registered [ 0.326397] acpiphp: Slot [3] registered [ 0.328180] acpiphp: Slot [4] registered [ 0.330127] acpiphp: Slot [11] registered [ 0.332200] acpiphp: Slot [12] registered [ 0.334128] acpiphp: Slot [13] registered [ 0.336169] acpiphp: Slot [14] registered [ 0.338131] acpiphp: Slot [15] registered [ 0.340124] acpiphp: Slot [16] registered [ 0.341124] acpiphp: Slot [17] registered [ 0.343136] acpiphp: Slot [18] registered [ 0.345138] acpiphp: Slot [19] registered [ 0.347167] acpiphp: Slot [20] registered [ 0.349135] acpiphp: Slot [21] registered [ 0.351136] acpiphp: Slot [22] registered [ 0.353124] acpiphp: Slot [23] registered [ 0.354171] acpiphp: Slot [24] registered [ 0.356324] acpiphp: Slot [25] registered [ 0.358299] acpiphp: Slot [26] registered [ 0.360122] acpiphp: Slot [27] registered [ 0.362138] acpiphp: Slot [28] registered [ 0.364136] acpiphp: Slot [29] registered [ 0.366132] acpiphp: Slot [30] registered [ 0.368161] acpiphp: Slot [31] registered [ 0.370140] PCI host bridge to bus 0000:00 [ 0.372032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.375030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.378029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.381033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.384031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.387056] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.390230] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.394227] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.397537] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.408945] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.414014] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.418024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.421022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.423020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.425755] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.430044] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.433048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.437950] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.443018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.459066] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.467020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.473517] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.483019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.494020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.515092] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.525296] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.533137] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.539017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.556019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.582736] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.594019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.603000] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.623017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.632052] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.639016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.647019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.663023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.685343] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.698020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.714000] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.751020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.770116] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.777023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.788022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.807018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.823000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.827472] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.832291] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.837966] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.851285] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.858837] iommu: Default domain type: Passthrough [ 0.864329] SCSI subsystem initialized [ 0.866579] ACPI: bus type USB registered [ 0.868245] usbcore: registered new interface driver usbfs [ 0.872114] usbcore: registered new interface driver hub [ 0.875112] usbcore: registered new device driver usb [ 0.877236] pps_core: LinuxPPS API ver. 1 registered [ 0.880015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.885082] PTP clock support registered [ 0.890117] EDAC MC: Ver: 3.0.0 [ 0.895187] PCI: Using ACPI for IRQ routing [ 0.903226] NetLabel: Initializing [ 0.906017] NetLabel: domain hash size = 128 [ 0.908013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.909357] NetLabel: unlabeled traffic allowed by default [ 0.913297] vgaarb: loaded [ 0.915427] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.916026] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.921333] clocksource: Switched to clocksource kvm-clock [ 1.098960] VFS: Disk quotas dquot_6.6.0 [ 1.103257] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.106874] *** VALIDATE ramfs *** [ 1.111067] *** VALIDATE hugetlbfs *** [ 1.114058] pnp: PnP ACPI init [ 1.117650] pnp: PnP ACPI: found 6 devices [ 1.137367] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.141444] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.143935] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.146407] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.149095] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.151867] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.154976] NET: Registered protocol family 2 [ 1.157924] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.163655] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.167768] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.173575] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.176934] TCP: Hash tables configured (established 65536 bind 65536) [ 1.179601] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.182827] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.186364] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.190544] NET: Registered protocol family 1 [ 1.193365] RPC: Registered named UNIX socket transport module. [ 1.195708] RPC: Registered udp transport module. [ 1.197921] RPC: Registered tcp transport module. [ 1.200142] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.203028] NET: Registered protocol family 44 [ 1.204971] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.207441] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.209447] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.211755] PCI: CLS 0 bytes, default 64 [ 1.213729] Unpacking initramfs... [ 3.617649] debug: unmapping init [mem 0xffff8eecbcc54000-0xffff8eecbffbffff] [ 3.624505] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.626902] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.632250] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.161396] Initialise system trusted keyrings [ 4.163549] Key type blacklist registered [ 4.165663] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.189094] zbud: loaded [ 4.200824] *** VALIDATE nfs *** [ 4.203300] *** VALIDATE nfs4 *** [ 4.206670] pstore: using deflate compression [ 4.216587] Platform Keyring initialized [ 4.374433] NET: Registered protocol family 38 [ 4.376870] Key type asymmetric registered [ 4.378432] Asymmetric key parser 'x509' registered [ 4.380995] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.384922] io scheduler mq-deadline registered [ 4.386949] io scheduler kyber registered [ 4.388692] io scheduler bfq registered [ 4.390968] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.394588] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.398133] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.401389] ACPI: Power Button [PWRF] [ 4.409410] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.417686] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.437273] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.444954] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.471796] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.501389] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.534262] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.539773] Non-volatile memory driver v1.3 [ 4.541835] Linux agpgart interface v0.103 [ 4.578329] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 4.582796] vda: detected capacity change from 0 to 74702848 [ 4.610297] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.615085] vdb: detected capacity change from 0 to 1073741824 [ 4.635392] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.638771] vdc: detected capacity change from 0 to 2621440000 [ 4.657369] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.659947] vdd: detected capacity change from 0 to 2621440000 [ 4.683589] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.686752] vde: detected capacity change from 0 to 4294967296 [ 4.706738] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.709887] vdf: detected capacity change from 0 to 4294967296 [ 4.721943] libphy: Fixed MDIO Bus: probed [ 4.733888] usbcore: registered new interface driver usbserial_generic [ 4.737623] usbserial: USB Serial support registered for generic [ 4.740922] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.746066] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.748616] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.751897] mousedev: PS/2 mouse device common for all mice [ 4.755675] rtc_cmos 00:05: RTC can wake from S4 [ 4.761571] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.765315] rtc_cmos 00:05: registered as rtc0 [ 4.770517] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.771330] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.780835] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.785402] intel_pstate: CPU model not supported [ 4.788431] hid: raw HID events driver (C) Jiri Kosina [ 4.799930] usbcore: registered new interface driver usbhid [ 4.802477] usbhid: USB HID core driver [ 4.809375] drop_monitor: Initializing network drop monitor service [ 4.815400] Initializing XFRM netlink socket [ 4.817939] NET: Registered protocol family 10 [ 4.821777] Segment Routing with IPv6 [ 4.825094] NET: Registered protocol family 17 [ 4.829362] mpls_gso: MPLS GSO support [ 4.840462] RAS: Correctable Errors collector initialized. [ 4.844936] AVX version of gcm_enc/dec engaged. [ 4.847992] AES CTR mode by8 optimization enabled [ 4.945838] sched_clock: Marking stable (4945754531, 0)->(6423300341, -1477545810) [ 4.949958] registered taskstats version 1 [ 4.952476] Loading compiled-in X.509 certificates [ 4.954976] zswap: loaded using pool lzo/zbud [ 5.036672] Key type big_key registered [ 5.050829] Key type encrypted registered [ 5.052835] ima: No TPM chip found, activating TPM-bypass! [ 5.056369] ima: Allocated hash algorithm: sha1 [ 5.059226] ima: No architecture policies found [ 5.061043] evm: Initialising EVM extended attributes: [ 5.063129] evm: security.selinux [ 5.064669] evm: security.ima [ 5.109236] evm: security.capability [ 5.134791] evm: HMAC attrs: 0x1 [ 5.138914] rtc_cmos 00:05: setting system clock to 2026-08-10 05:36:21 UTC (1786340181) [ 5.152655] debug: unmapping init [mem 0xffffffff8cc03000-0xffffffff8cdfffff] [ 5.158390] debug: unmapping init [mem 0xffffffff8b982000-0xffffffff8bc58fff] [ 5.166373] Write protecting the kernel read-only data: 28672k [ 5.172506] debug: unmapping init [mem 0xffffffff8a003000-0xffffffff8a1fffff] [ 5.177201] debug: unmapping init [mem 0xffffffff8a914000-0xffffffff8a9fffff] [ 5.231753] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.241519] systemd[1]: Detected virtualization kvm. [ 5.244578] systemd[1]: Detected architecture x86-64. [ 5.247274] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.287568] systemd[1]: No hostname configured. [ 5.290710] systemd[1]: Set hostname to . [ 5.294765] random: systemd: uninitialized urandom read (16 bytes read) [ 5.298770] systemd[1]: Initializing machine ID from random generator. [ 5.408714] random: ln: uninitialized urandom read (6 bytes read) [ 5.568745] random: systemd: uninitialized urandom read (16 bytes read) [ 5.571471] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 5.575722] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.583399] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.039952] device-mapper: uevent: version 1.0.3 [ 7.042923] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.324443] random: fast init done [ 8.381969] virtio_net virtio0 ens2: renamed from eth0 [ 8.474172] scsi host0: ata_piix [ 8.499219] scsi host1: ata_piix [ 8.502290] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.507539] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.646541] random: crng init done [ 13.649361] random: 7 urandom warning(s) missed due to ratelimiting [ 14.378692] dracut-initqueue[594]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.866724] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.195251] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.071588] SELinux: Disabled at runtime. [ 19.155652] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.168642] systemd[1]: Detected virtualization kvm. [ 19.172889] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.720340] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.734201] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.746019] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.760185] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.765173] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.776293] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.784293] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ 21.128544] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 22.206967] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.049630] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.121194] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.215648] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.323407] EDAC sbridge: Ver: 1.1.2 [ 26.947984] Key type dns_resolver registered [ 27.395753] NFS: Registering the id_resolver key type [ 27.398126] Key type id_resolver registered [ 27.399437] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg328-server login: [ 46.225097] hrtimer: interrupt took 3021136 ns [ 78.461987] libcfs: loading out-of-tree module taints kernel. [ 78.488019] Key type ._llcrypt registered [ 78.489804] Key type .llcrypt registered [ 78.564775] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 95.142647] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 96.979491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 96.999368] alg: No test for adler32 (adler32-zlib) [ 98.435315] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 98.982575] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 100.704702] Key type lgssc registered [ 102.061397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 117.489122] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 151.842499] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 163.723486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 163.750472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.943390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 164.962641] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 165.023132] Lustre: lustre-MDT0000: new disk, initializing [ 165.097291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 165.115493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 168.796399] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.386206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 180.459802] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 180.490577] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 180.495282] Lustre: Skipped 1 previous similar message [ 180.548771] Lustre: lustre-MDT0001: new disk, initializing [ 180.598248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 180.622638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 180.636516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 184.243940] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.491856] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 196.473175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 196.729307] Lustre: lustre-OST0000: new disk, initializing [ 196.734287] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 196.738600] Lustre: 8449:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 196.798747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.044700] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.265900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 202.284870] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 202.303223] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 214.157413] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 214.246933] Lustre: lustre-OST0001: new disk, initializing [ 214.249903] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 214.253990] Lustre: 9521:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 214.307276] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 219.734741] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.188298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 220.196319] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 220.266251] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 230.654784] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 236.657019] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 241.599709] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing check_logdir /tmp/testlogs/ [ 245.557607] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing yml_node [ 249.981713] Lustre: DEBUG MARKER: Client: 2.17.56.55 [ 251.896909] Lustre: DEBUG MARKER: MDS: 2.17.56.55 [ 253.985759] Lustre: DEBUG MARKER: OSS: 2.17.56.55 [ 255.214081] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 10 01:40:29 EDT 2026 [ 268.988961] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 269.977557] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 287.200851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 287.202464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 287.208938] Lustre: Skipped 1 previous similar message [ 290.748908] Lustre: server umount lustre-MDT0000 complete [ 293.704068] LustreError: 8459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786340470 with bad export cookie 4887705502805714045 [ 293.711595] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 293.714834] LustreError: 8459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 297.441572] LustreError: 10355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.443594] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.444586] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 297.444591] Lustre: Skipped 3 previous similar messages [ 297.459515] LustreError: 10355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 297.519260] Lustre: Skipped 3 previous similar messages [ 298.466674] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.467363] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 298.494889] Lustre: Skipped 2 previous similar messages [ 298.497466] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 303.588870] LustreError: 9529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 303.588982] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 303.606561] LustreError: 9529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 303.633283] Lustre: Skipped 1 previous similar message [ 308.191140] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 308.481190] Lustre: server umount lustre-MDT0001 complete [ 311.107516] Lustre: server umount lustre-OST0000 complete [ 314.038434] Lustre: server umount lustre-OST0001 complete [ 319.945789] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 328.413150] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 360.585456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 360.737332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 360.759250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 360.814707] Lustre: lustre-MDT0000: new disk, initializing [ 360.869789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 360.889445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 364.683958] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 374.229805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 374.318405] Lustre: 15069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 374.364815] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 374.368114] Lustre: Skipped 1 previous similar message [ 374.422247] Lustre: lustre-MDT0001: new disk, initializing [ 374.482573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 374.553288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 374.584875] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 378.873688] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 387.972768] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 394.377645] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 400.520889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 400.761450] Lustre: lustre-OST0000: new disk, initializing [ 400.764960] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 400.769957] Lustre: 17035:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 400.854619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 402.132957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 402.140480] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 402.259161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 406.090285] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 414.862401] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 419.256751] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 419.437098] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 422.956936] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 423.139353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 428.001892] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 428.002370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 428.011924] LustreError: Skipped 1 previous similar message [ 428.063463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 428.068800] Lustre: Skipped 3 previous similar messages [ 430.939719] Lustre: server umount lustre-OST0000 complete [ 436.193416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 436.200963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 436.213147] Lustre: Skipped 1 previous similar message [ 436.220493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 440.305673] Lustre: server umount lustre-MDT0000 complete [ 443.375525] LustreError: 15080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 443.387923] LustreError: 15080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 444.056036] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786340620 with bad export cookie 4887705502805715578 [ 444.063603] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 444.065080] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 444.493829] Lustre: server umount lustre-MDT0001 complete [ 449.661514] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:43:43 (1786340623) [ 467.372953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 467.689064] LustreError: 19911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 467.733423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 471.069455] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 478.183331] LustreError: 19912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 478.222205] LustreError: 19912:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 478.995131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.312338] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 490.718481] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 496.623285] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 498.903201] Lustre: Failing over lustre-MDT0000 [ 499.099836] Lustre: server umount lustre-MDT0000 complete [ 500.192711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 500.200447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.213166] Lustre: Skipped 1 previous similar message [ 500.220474] LustreError: 20639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.397177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 518.476963] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 523.520833] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 523.763535] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 523.784720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 523.802810] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 531.960432] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 539.108864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.110120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 539.110202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 539.110207] Lustre: Skipped 2 previous similar messages [ 539.116746] Lustre: Skipped 2 previous similar messages [ 541.789239] Lustre: server umount lustre-MDT0000 complete [ 544.227441] LustreError: 19906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.244957] LustreError: 19906:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 544.733503] LustreError: 19893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786340721 with bad export cookie 4887705502805716992 [ 544.751165] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 544.956800] Lustre: server umount lustre-MDT0001 complete [ 551.107391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 551.462344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 551.467314] Lustre: Skipped 2 previous similar messages [ 554.838961] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 561.419776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 565.596822] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 571.253312] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 575.840609] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 577.830809] Lustre: Failing over lustre-MDT0000 [ 578.184071] Lustre: server umount lustre-MDT0000 complete [ 581.608101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 581.623604] Lustre: Skipped 1 previous similar message [ 595.195965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 595.272563] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 599.241715] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 600.546411] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 600.550786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 600.555061] Lustre: Skipped 1 previous similar message [ 600.564050] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 608.249406] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 612.837211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 612.846122] Lustre: Skipped 2 previous similar messages [ 627.167253] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 628.192485] LustreError: 23971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 628.214210] LustreError: 23971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 629.392252] Lustre: server umount lustre-MDT0000 complete [ 633.169802] LustreError: 23226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786340809 with bad export cookie 4887705502805717930 [ 633.176864] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 633.179940] LustreError: 23226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 633.419392] Lustre: server umount lustre-MDT0001 complete [ 644.224595] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:46:58 (1786340818) [ 664.351713] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 676.271549] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 714.061594] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 724.609790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 724.883501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 724.903791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 724.961324] Lustre: lustre-MDT0000: new disk, initializing [ 725.010191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 725.015852] Lustre: Skipped 2 previous similar messages [ 725.024793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 729.496360] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 740.535231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 740.673602] Lustre: 29619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 740.908176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 746.959673] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 756.430993] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 762.944443] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 770.600205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 770.855288] Lustre: 31588:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 772.810676] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 772.860706] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 773.002201] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 773.019279] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 776.636408] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 786.921297] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 793.862091] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 814.051068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 814.051960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 814.073871] LustreError: Skipped 1 previous similar message [ 814.091068] Lustre: Skipped 2 previous similar messages [ 814.093840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 814.118063] Lustre: Skipped 5 previous similar messages [ 816.817301] Lustre: server umount lustre-OST0000 complete [ 823.288394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 826.387739] Lustre: server umount lustre-MDT0000 complete [ 829.417074] LustreError: 30495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 829.426735] LustreError: 30495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 830.462555] LustreError: 29611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786341006 with bad export cookie 4887705502805718336 [ 830.477390] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 830.478729] LustreError: 29611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 830.857776] Lustre: server umount lustre-MDT0001 complete [ 842.102847] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 844.291566] Key type lgssc unregistered [ 844.570754] LNet: 33568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.575554] LNetError: 33568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 845.605646] LNet: Removed LNI 192.168.203.128@tcp [ 846.356257] Key type .llcrypt unregistered [ 846.357956] Key type ._llcrypt unregistered [ 870.959875] Key type ._llcrypt registered [ 870.981964] Key type .llcrypt registered [ 871.104519] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 873.247722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 873.274219] alg: No test for adler32 (adler32-zlib) [ 874.285889] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 874.489441] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 876.207199] Key type lgssc registered [ 877.192596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 883.656957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 883.671952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 884.995286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 889.499515] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 895.461610] LustreError: 34984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 895.494233] LustreError: 34984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 898.135536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 899.004268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 904.284973] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 911.406886] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 916.450823] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 923.735334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 924.183801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 925.129650] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 930.210110] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 937.382025] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 952.236577] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:52:06 (1786341126) [ 954.393385] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 956.175804] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 962.534808] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:52:16 (1786341136) [ 971.409393] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:52:25 (1786341145) [ 973.424839] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 984.588915] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 984.592780] Lustre: Skipped 2 previous similar messages [ 986.159213] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:52:40 (1786341160) [ 990.130663] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 990.140600] Lustre: Skipped 3 previous similar messages [ 1008.067263] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1008.074094] Lustre: Skipped 6 previous similar messages [ 1010.634872] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:53:05 (1786341185) [ 1012.817570] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1016.785435] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1025.941748] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:53:19 (1786341199) [ 1028.073876] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1028.094642] Lustre: Skipped 3 previous similar messages [ 1040.267441] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:53:34 (1786341214) [ 1053.157771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1053.175747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1053.186815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1053.198900] Lustre: Skipped 1 previous similar message [ 1053.216463] Lustre: Skipped 1 previous similar message [ 1056.260965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1056.287845] Lustre: Skipped 1 previous similar message [ 1059.298205] Lustre: server umount lustre-OST0000 complete [ 1063.395255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1063.401294] LustreError: Skipped 1 previous similar message [ 1063.403333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1063.410684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1066.464204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1066.465755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1068.702637] Lustre: server umount lustre-MDT0000 complete [ 1071.584558] LustreError: 37387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1071.602841] LustreError: 37387:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1071.710611] LustreError: 34963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786341248 with bad export cookie 12736569802424772017 [ 1071.716533] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1071.721348] LustreError: 34963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1072.051080] Lustre: server umount lustre-MDT0001 complete [ 1081.834819] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1084.661761] Key type lgssc unregistered [ 1084.874886] LNet: 42842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1084.881344] LNetError: 42842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1084.969018] LNet: Removed LNI 192.168.203.128@tcp [ 1085.575916] Key type .llcrypt unregistered [ 1085.578063] Key type ._llcrypt unregistered [ 1103.032778] Key type ._llcrypt registered [ 1103.034993] Key type .llcrypt registered [ 1103.120681] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 1116.947839] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1118.433929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1118.503272] alg: No test for adler32 (adler32-zlib) [ 1119.516314] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 1119.667024] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1121.303202] Key type lgssc registered [ 1122.502787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1154.023140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1154.091315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1155.304767] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1155.332416] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1155.388172] Lustre: lustre-MDT0000: new disk, initializing [ 1155.437730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1155.461896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1159.564101] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1168.481225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1168.650028] Lustre: 45998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1168.698290] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1168.701903] Lustre: Skipped 1 previous similar message [ 1168.760886] Lustre: lustre-MDT0001: new disk, initializing [ 1168.869342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1168.921401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1168.939780] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1172.797929] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1181.051980] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1186.230912] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1192.832797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1192.997770] Lustre: lustre-OST0000: new disk, initializing [ 1193.001785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1193.009286] Lustre: 47964:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1193.100226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1198.527297] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1200.157401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1200.168769] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1200.286104] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1206.181590] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1210.148234] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1210.370210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1214.644667] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1214.819277] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1220.575771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1220.580261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1220.588167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1221.089306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1221.094290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1221.112807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1222.479463] Lustre: server umount lustre-OST0000 complete [ 1226.208897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1226.216784] Lustre: Skipped 1 previous similar message [ 1226.218991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1226.227538] Lustre: Skipped 2 previous similar messages [ 1231.328598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1232.099777] Lustre: server umount lustre-MDT0000 complete [ 1234.658061] LustreError: 47963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786341411 with bad export cookie 18111778233760546148 [ 1234.666748] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1234.678560] LustreError: 47963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1234.975152] Lustre: server umount lustre-MDT0001 complete [ 1241.760856] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:56:56 (1786341416) [ 1247.343021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1247.753511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1251.588912] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1252.833603] LustreError: 50423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1252.844960] LustreError: 50423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1257.608808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1257.817632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1261.252368] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1266.622336] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1270.382936] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1275.896696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1276.116846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1280.791325] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1286.383309] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1299.676546] LustreError: 52972:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1300.323447] Lustre: Modifying parameter general.timeout=1 in log params [ 1301.465290] Lustre: Modifying parameter general.timeout=3 in log params [ 1301.474222] Lustre: Skipped 1 previous similar message [ 1303.660557] Lustre: Modifying parameter general.timeout=6 in log params [ 1303.664430] Lustre: Skipped 2 previous similar messages [ 1307.709787] Lustre: Modifying parameter general.timeout=12 in log params [ 1307.715179] Lustre: Skipped 5 previous similar messages [ 1316.009225] Lustre: Modifying parameter general.timeout=25 in log params [ 1316.013282] Lustre: Skipped 12 previous similar messages [ 1332.140636] Lustre: Modifying parameter general.timeout=47 in log params [ 1332.144621] Lustre: Skipped 21 previous similar messages [ 1364.492146] Lustre: Modifying parameter general.timeout=101 in log params [ 1364.494371] Lustre: Skipped 53 previous similar messages [ 1386.331846] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:59:21 (1786341561) [ 1387.686946] Lustre: 59891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1389.600523] Lustre: 60088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1389.611510] Lustre: 60088:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1392.164595] Lustre: 60382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1392.171129] Lustre: 60382:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1405.892522] Lustre: 61731:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1405.903592] LustreError: 61731:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1407.926156] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:59:42 (1786341582) [ 1416.672832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1416.679772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1416.690551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1416.693795] Lustre: Skipped 1 previous similar message [ 1419.901027] Lustre: server umount lustre-MDT0000 complete [ 1422.077612] LustreError: 50405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786341598 with bad export cookie 18111778233760547030 [ 1422.087114] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1422.308125] Lustre: server umount lustre-MDT0001 complete [ 1434.786972] Lustre: server umount lustre-OST0000 complete [ 1450.285071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1450.437918] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1450.458439] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1450.468860] Lustre: Skipped 51 previous similar messages [ 1450.604340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1453.126471] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1458.030933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1458.088271] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1458.107216] Lustre: 64259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1461.211395] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1465.821205] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1469.178808] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1473.360226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1473.463329] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1477.238355] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1481.613273] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1482.626199] Lustre: 66825:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1489.892176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1489.893425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1489.895796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1489.898807] Lustre: Skipped 3 previous similar messages [ 1489.916752] Lustre: Skipped 4 previous similar messages [ 1492.450453] Lustre: server umount lustre-OST0000 complete [ 1500.829821] Lustre: server umount lustre-MDT0000 complete [ 1502.910846] LustreError: 64253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786341679 with bad export cookie 18111778233760549641 [ 1502.916419] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1502.918039] LustreError: 64253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1503.119911] Lustre: server umount lustre-MDT0001 complete [ 1509.881644] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1511.742167] Key type lgssc unregistered [ 1511.941749] LNet: 68048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1511.948612] LNetError: 68048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1511.965183] LNet: Removed LNI 192.168.203.128@tcp [ 1512.555167] Key type .llcrypt unregistered [ 1512.557333] Key type ._llcrypt unregistered [ 1527.095299] Key type ._llcrypt registered [ 1527.097229] Key type .llcrypt registered [ 1527.173290] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1527.672582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1527.727114] alg: No test for adler32 (adler32-zlib) [ 1528.697678] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 1528.818300] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1530.439633] Key type lgssc registered [ 1531.078867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1535.230849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1535.240828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1536.484435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1538.748152] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1543.687166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1544.126519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1546.688538] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1550.735727] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1553.859386] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1557.240141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1557.532082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1560.558662] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1564.600355] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1574.402712] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1578.976455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1578.978716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1578.985684] Lustre: Skipped 2 previous similar messages [ 1578.989289] Lustre: Skipped 1 previous similar message [ 1581.051403] Lustre: server umount lustre-MDT0000 complete [ 1583.248677] LustreError: 72123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786341759 with bad export cookie 16393668098324310666 [ 1583.249941] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1583.257234] LustreError: 72123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1583.274162] LustreError: 69432:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 1584.096880] LustreError: 69434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1584.097390] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1584.106396] LustreError: 69434:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1584.123399] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1585.459881] Lustre: server umount lustre-MDT0001 complete [ 1587.795400] Lustre: server umount lustre-OST0000 complete [ 1603.793579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1603.937973] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1603.954344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1604.108609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1606.615466] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1611.797574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1611.858952] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1611.880121] Lustre: 74301:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1612.162949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1614.872337] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1619.238431] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1622.331854] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1626.239949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1626.368646] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1626.645396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1630.452707] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1635.018107] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1636.005686] Lustre: 76877:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1637.021177] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1644.517911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1644.520165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1644.520832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1644.525367] Lustre: Skipped 1 previous similar message [ 1645.694146] Lustre: server umount lustre-OST0000 complete [ 1648.096973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1648.102821] LustreError: Skipped 1 previous similar message [ 1648.105983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1648.114537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1648.117175] Lustre: Skipped 1 previous similar message [ 1654.094646] Lustre: server umount lustre-MDT0000 complete [ 1654.752237] LustreError: 74309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1654.761202] LustreError: 74309:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1656.123441] LustreError: 74294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786341832 with bad export cookie 16393668098324311457 [ 1656.131184] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1656.133722] LustreError: 74294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1656.327693] Lustre: server umount lustre-MDT0001 complete [ 1662.256801] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1663.820264] Key type lgssc unregistered [ 1663.977585] LNet: 78087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1663.981635] LNetError: 78087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1664.997521] LNet: Removed LNI 192.168.203.128@tcp [ 1665.522402] Key type .llcrypt unregistered [ 1665.524718] Key type ._llcrypt unregistered [ 1680.205778] Key type ._llcrypt registered [ 1680.208260] Key type .llcrypt registered [ 1680.295789] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1681.196613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1681.340923] alg: No test for adler32 (adler32-zlib) [ 1682.288908] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 1682.446440] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1684.079165] Key type lgssc registered [ 1684.811274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1689.803442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1689.821204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1691.105730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1693.822219] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1698.555259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1698.883385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1701.268796] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1705.242678] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1708.196860] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1711.827528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1712.167830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1715.428859] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1719.554564] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1727.026845] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1727.532677] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1728.890505] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1728.895533] Lustre: Skipped 2 previous similar messages [ 1731.099456] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1731.104364] Lustre: Skipped 4 previous similar messages [ 1735.130915] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1735.134213] Lustre: Skipped 10 previous similar messages [ 1743.234853] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1743.239809] Lustre: Skipped 17 previous similar messages [ 1759.520459] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1759.528187] Lustre: Skipped 36 previous similar messages [ 1774.591744] Lustre: 87553:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1776.155545] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 02:05:51 (1786341951) [ 1776.979212] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1777.930610] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 02:05:52 (1786341952) [ 1780.707194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1780.708268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.710288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1780.722581] Lustre: Skipped 1 previous similar message [ 1785.826370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1785.830217] Lustre: Skipped 2 previous similar messages [ 1790.949163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1790.952488] Lustre: Skipped 1 previous similar message [ 1794.016130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1794.164813] Lustre: server umount lustre-OST0000 complete [ 1801.185188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1801.185960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1801.186636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1801.198438] Lustre: Skipped 1 previous similar message [ 1802.736102] Lustre: server umount lustre-MDT0000 complete [ 1804.419916] LustreError: 79459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786341980 with bad export cookie 17504496543552310780 [ 1804.420444] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1804.426441] LustreError: 79459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1804.571953] Lustre: server umount lustre-MDT0001 complete [ 1809.478667] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1811.054842] Key type lgssc unregistered [ 1811.225719] LNet: 88807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1811.231315] LNetError: 88807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1811.245881] LNet: Removed LNI 192.168.203.128@tcp [ 1811.724374] Key type .llcrypt unregistered [ 1811.726555] Key type ._llcrypt unregistered [ 1813.949363] Key type ._llcrypt registered [ 1813.951016] Key type .llcrypt registered [ 1814.011733] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1816.254640] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules [ 1816.711514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1816.739195] alg: No test for adler32 (adler32-zlib) [ 1817.706223] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 1817.849770] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1819.463159] Key type lgssc registered [ 1820.017705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1820.630920] LustreError: 89961:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 1830.460690] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1835.208675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1841.373401] LustreError: 89961:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1841.389978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1842.653104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1844.418438] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1848.909488] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 02:07:03 (1786342023) [ 1851.127169] LustreError: 91015:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1851.132662] LustreError: 91015:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1851.137268] LustreError: 91015:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 1851.315828] Lustre: server umount lustre-MDT0000 complete [ 1857.286497] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1858.732615] Key type lgssc unregistered [ 1858.898493] LNet: 92511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1858.904245] LNetError: 92511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1858.912314] LNet: Removed LNI 192.168.203.128@tcp [ 1859.345331] Key type .llcrypt unregistered [ 1859.347145] Key type ._llcrypt unregistered [ 1869.072270] Key type ._llcrypt registered [ 1869.074615] Key type .llcrypt registered [ 1869.132823] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1869.744378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1869.757810] alg: No test for adler32 (adler32-zlib) [ 1870.696414] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 1870.808972] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1872.407381] Key type lgssc registered [ 1873.065847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1877.130346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1877.143456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1878.421788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1880.716532] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1885.115155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1885.476489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1887.942803] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1891.712099] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1894.650496] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1898.050146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1898.397760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1901.541369] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1905.208243] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1937.329544] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 02:08:32 (1786342112) [ 1938.048089] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1938.881477] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 02:08:33 (1786342113) [ 1940.450510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1940.451581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1940.457072] Lustre: Skipped 1 previous similar message [ 1945.569709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1945.572868] Lustre: Skipped 4 previous similar messages [ 1946.509408] Lustre: server umount lustre-MDT0000 complete [ 1948.150432] LustreError: 93687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786342124 with bad export cookie 2960866809684498754 [ 1948.153248] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1948.157782] LustreError: 93687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1948.283184] Lustre: server umount lustre-MDT0001 complete [ 1960.161098] Lustre: server umount lustre-OST0000 complete [ 1965.946052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1966.186358] LustreError: 97974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1966.198170] LustreError: 97974:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1966.281883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1968.162097] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1971.680466] LustreError: 97975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1971.914610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1972.246853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1974.115722] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1977.223041] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1979.796531] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1986.087871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1986.186178] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1986.190089] LustreError: 97964:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1986.200310] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1986.205264] LustreError: 100054:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1986.209413] LustreError: 100054:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1986.212771] LustreError: 100054:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1986.253367] Lustre: server umount lustre-OST0000 complete [ 1986.256354] LustreError: 100054:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1990.488510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1990.588754] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1990.593212] LustreError: 97964:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1990.599258] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1990.604203] LustreError: 100516:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1990.607790] LustreError: 100516:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1990.610982] LustreError: 100516:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1990.642742] Lustre: server umount lustre-OST0000 complete [ 1990.646126] LustreError: 100516:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1995.521568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1995.638463] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1995.645623] Lustre: Found index 0 for lustre-OST0000, updating log [ 1995.649689] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1995.708471] Lustre: lustre-OST0000: new disk, initializing [ 1995.711040] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1995.715475] Lustre: 101051:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1995.872622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1998.854560] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2000.759758] Lustre: Failing over lustre-OST0000 [ 2000.838955] Lustre: server umount lustre-OST0000 complete [ 2002.912721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2002.917512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2002.924051] Lustre: Skipped 1 previous similar message [ 2002.926605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2007.521980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2007.522171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2008.811452] Lustre: server umount lustre-MDT0000 complete [ 2010.780772] LustreError: 97956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786342187 with bad export cookie 2960866809684501484 [ 2010.785331] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2010.790857] LustreError: 97956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2021.426331] Lustre: server umount lustre-MDT0001 complete [ 2025.699615] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 02:10:00 (1786342200) [ 2033.263214] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2037.640736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2037.798206] LustreError: 103391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2037.831875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2039.457334] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2042.848186] LustreError: 103392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2043.002994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2045.052143] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2046.197185] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2048.879043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2049.228785] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2051.519195] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2053.346668] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2053.357389] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2053.357553] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2054.986284] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2055.027289] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2055.068658] Lustre: lustre-OST0001: new disk, initializing [ 2055.070601] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2055.074414] Lustre: 105711:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2056.345546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2056.349834] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2056.389169] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2057.423399] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2060.821271] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2066.876851] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2068.960664] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2068.963192] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2068.964995] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2068.968182] Lustre: Skipped 2 previous similar messages [ 2068.973435] Lustre: Skipped 3 previous similar messages [ 2074.853794] Lustre: server umount lustre-MDT0001 complete [ 2078.827844] LustreError: 103386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2079.022539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2079.080068] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2079.082054] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2079.100825] Lustre: 103382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2079.426956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2079.432697] Lustre: Skipped 3 previous similar messages [ 2080.297089] LustreError: 107321:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2080.302270] LustreError: 107321:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2080.306641] LustreError: 103384:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2080.352809] LustreError: 107332:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.128@tcp: cfg command failed: rc = -17 [ 2080.358381] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2080.358381] [ 2081.182863] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2084.839554] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2085.240890] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 02:11:00 (1786342260) [ 2092.683426] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2101.013429] Lustre: 109316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2111.825308] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2113.397901] Lustre: 110449:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2171.360394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2171.365301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2171.372443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2171.377133] Lustre: Skipped 4 previous similar messages [ 2175.248812] Lustre: server umount lustre-MDT0000 complete [ 2176.993401] LustreError: 105473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2177.001208] LustreError: 105473:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2177.161688] LustreError: 103374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786342353 with bad export cookie 2960866809684502359 [ 2177.165866] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2177.167626] LustreError: 103374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2177.339803] Lustre: server umount lustre-MDT0001 complete [ 2189.323649] Lustre: server umount lustre-OST0000 complete [ 2201.512229] Lustre: server umount lustre-OST0001 complete [ 2204.165758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2209.116870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2213.334842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2218.083986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2227.128744] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2232.089449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2232.099499] Lustre: lustre-MDT0000: reset Object Index mappings [ 2232.297504] LustreError: 114397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2232.337923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2234.380308] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2238.726070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2238.739569] Lustre: lustre-MDT0001: reset Object Index mappings [ 2241.351723] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2242.894711] Lustre: 115546:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2246.131674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2249.348419] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2249.578112] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:231 to 0x280000400:257) [ 2253.365523] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2256.750663] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2256.751504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 2256.784366] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2256.809478] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 2261.163981] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2262.987378] Lustre: 117419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2287.497426] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 02:14:22 (1786342462) [ 2318.311521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2318.313925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2318.318083] Lustre: Skipped 5 previous similar messages [ 2318.324246] Lustre: Skipped 4 previous similar messages [ 2322.586443] Lustre: server umount lustre-MDT0000 complete [ 2323.427180] LustreError: 117768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.435857] LustreError: 117768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2324.379380] LustreError: 114379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786342500 with bad export cookie 2960866809684549126 [ 2324.383792] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2324.388825] LustreError: 114379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2330.732224] Lustre: server umount lustre-MDT0001 complete [ 2338.837845] Lustre: server umount lustre-OST0000 complete [ 2346.677620] Lustre: server umount lustre-OST0001 complete [ 2349.569367] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 2354.253119] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2370.836262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2370.943603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2370.960597] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2371.001490] Lustre: lustre-MDT0000: new disk, initializing [ 2371.042241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2371.045569] Lustre: Skipped 3 previous similar messages [ 2371.053798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2372.935467] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2376.493941] Lustre: server umount lustre-MDT0000 complete [ 2380.710757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2380.846763] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2380.856282] Lustre: 121677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2380.864573] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2382.739501] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2384.897259] Lustre: server umount lustre-MDT0000 complete [ 2388.373936] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 02:16:03 (1786342563) [ 2389.092398] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2389.888568] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 02:16:05 (1786342565) [ 2396.989609] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2401.457713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2403.549356] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2406.933319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2406.989616] Lustre: 123725:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2407.023791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2407.026288] Lustre: Skipped 1 previous similar message [ 2407.067580] Lustre: lustre-MDT0001: new disk, initializing [ 2407.127087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2407.131865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2408.993994] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2411.838705] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2414.695963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2414.842518] Lustre: lustre-OST0000: new disk, initializing [ 2414.845177] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2414.849410] Lustre: 125302:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2416.517171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2416.521441] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2416.577266] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2417.581640] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2423.015232] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2423.100388] Lustre: lustre-OST0001: new disk, initializing [ 2423.103251] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2423.107139] Lustre: 126171:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2425.037144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2425.043116] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2425.098718] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2425.799605] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2431.147626] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2432.837682] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2595.945506] Lustre: 132548:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2595.950088] Lustre: 132548:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2595.953623] Lustre: 132548:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2595.956592] Lustre: 132548:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2595.959781] Lustre: 132548:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2595.963300] Lustre: 132548:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2705.792604] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:21:20 (1786342880) [ 2719.616765] Lustre: server umount lustre-MDT0000 complete [ 2719.711899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2719.716203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2719.721691] Lustre: Skipped 3 previous similar messages [ 2719.723748] LustreError: 125297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2719.729376] LustreError: 125297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2720.912885] LustreError: 123717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786342897 with bad export cookie 2960866809684608388 [ 2720.919346] LustreError: 123717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2720.923160] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2722.273930] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2722.276554] Lustre: Skipped 7 previous similar messages [ 2727.121102] Lustre: server umount lustre-MDT0001 complete [ 2734.593069] Lustre: server umount lustre-OST0000 complete [ 2741.978602] Lustre: server umount lustre-OST0001 complete [ 2744.231810] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 2747.606780] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2760.086809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2760.172899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2760.187563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2760.226565] Lustre: lustre-MDT0000: new disk, initializing [ 2760.249520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2760.252063] Lustre: Skipped 5 previous similar messages [ 2760.258865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2761.753661] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2766.330450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2766.384097] Lustre: 135565:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2766.511413] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2768.177657] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2772.204158] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2774.209884] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2776.781982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2776.920211] Lustre: 137534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2778.514580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2778.519575] Lustre: Skipped 1 previous similar message [ 2778.521702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2778.569426] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2779.491748] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2783.724145] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2787.618901] Lustre: lustre-MDD0000: changelog on [ 2788.761894] Lustre: lustre-MDD0001: changelog on [ 3296.860355] Lustre: lustre-MDD0001: changelog off [ 3297.526156] Lustre: 135572:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 3297.853521] Lustre: lustre-MDD0000: changelog off [ 3300.832514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3300.834376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3300.835711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3300.835716] Lustre: Skipped 2 previous similar messages [ 3300.836282] LustreError: Skipped 1 previous similar message [ 3300.841432] Lustre: Skipped 5 previous similar messages [ 3305.625815] Lustre: server umount lustre-OST0000 complete [ 3309.535889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3321.311121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3321.431956] Lustre: server umount lustre-MDT0000 complete [ 3322.795716] LustreError: 137533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786343499 with bad export cookie 2960866809685413038 [ 3322.800599] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3322.803134] LustreError: 137533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3322.944432] Lustre: server umount lustre-MDT0001 complete [ 3326.887263] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 3328.005549] Key type lgssc unregistered [ 3328.145447] LNet: 146212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3328.149312] LNetError: 146212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3328.159331] LNet: Removed LNI 192.168.203.128@tcp [ 3328.500119] Key type .llcrypt unregistered [ 3328.501747] Key type ._llcrypt unregistered [ 3335.009134] Key type ._llcrypt registered [ 3335.010293] Key type .llcrypt registered [ 3335.042882] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 3341.074317] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3341.390845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3341.434617] alg: No test for adler32 (adler32-zlib) [ 3342.293726] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 3342.378553] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 3343.959164] Key type lgssc registered [ 3344.355798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3356.218343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3356.226402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3357.323781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3357.336120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3357.372883] Lustre: lustre-MDT0000: new disk, initializing [ 3357.394409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3357.401196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3358.845822] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3363.357123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3363.406237] Lustre: 149384:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3363.434170] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3363.436814] Lustre: Skipped 1 previous similar message [ 3363.475536] Lustre: lustre-MDT0001: new disk, initializing [ 3363.513991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3363.529614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3363.533872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3365.084229] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3369.126651] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3371.162118] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3373.647504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3373.784531] Lustre: lustre-OST0000: new disk, initializing [ 3373.786502] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3373.788919] Lustre: 151351:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3373.819314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3375.920267] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3378.681889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3378.685446] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3378.718022] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3379.559704] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3381.760943] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_module obdecho/obdecho [ 3382.361731] LustreError: 152355:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3382.365252] LustreError: 152355:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 3383.001921] LustreError: 152452:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 3388.898230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3388.900196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3388.903454] Lustre: Skipped 2 previous similar messages [ 3388.906234] Lustre: Skipped 1 previous similar message [ 3392.482163] Lustre: server umount lustre-MDT0000 complete [ 3393.980508] LustreError: 149376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786343570 with bad export cookie 4169655660466543048 [ 3393.985351] LustreError: 149376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3393.986201] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3394.017083] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3394.023122] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3394.126021] Lustre: server umount lustre-MDT0001 complete [ 3401.818653] Lustre: server umount lustre-OST0000 complete [ 3408.942062] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 3412.768407] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3425.478746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3425.573976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3425.587483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3425.626992] Lustre: lustre-MDT0000: new disk, initializing [ 3425.649175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3425.655331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3427.009251] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3431.181643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3431.222988] Lustre: 156157:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3431.247143] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3431.249105] Lustre: Skipped 1 previous similar message [ 3431.284928] Lustre: lustre-MDT0001: new disk, initializing [ 3431.319481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3431.331689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3431.335052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3432.697943] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3436.344673] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3438.084162] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3440.257348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3440.363141] Lustre: lustre-OST0000: new disk, initializing [ 3440.365652] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3440.369385] Lustre: 158126:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3440.397171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3442.280579] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3445.866206] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3447.323945] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3449.846525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3449.849856] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3449.888271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3450.455580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3452.016433] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3452.103516] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3454.945744] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3454.947414] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3454.949721] LustreError: Skipped 1 previous similar message [ 3454.954654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3454.957829] Lustre: Skipped 1 previous similar message [ 3458.899106] Lustre: server umount lustre-OST0000 complete [ 3462.111858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3462.114059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3462.117781] Lustre: Skipped 1 previous similar message [ 3462.119350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3466.468148] Lustre: server umount lustre-MDT0000 complete [ 3467.816832] LustreError: 156149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786343644 with bad export cookie 4169655660466544007 [ 3467.820405] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3467.823119] LustreError: 156149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3467.958149] Lustre: server umount lustre-MDT0001 complete [ 3469.282979] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:34:04 (1786343644) [ 3469.808633] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3470.404042] Lustre: DEBUG MARKER: == conf-sanity test 138: MDT mount failure must not LBUG destroying recovery exports ========================================================== 02:34:05 (1786343645) [ 3476.655520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3476.826968] LustreError: 161113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3476.833549] LustreError: 161113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3476.848906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3478.217857] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3481.144843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3482.653364] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3485.109052] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3486.865233] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3489.079262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3491.144174] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3493.600019] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3506.736241] Lustre: *** cfs_fail_loc=718, val=0*** [ 3507.168049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3507.171435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3507.175820] Lustre: Skipped 1 previous similar message [ 3507.177899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3507.180287] Lustre: Skipped 2 previous similar messages [ 3512.983916] Lustre: server umount lustre-MDT0000 complete [ 3513.825516] LustreError: 162555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3513.832662] LustreError: 162555:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3514.452638] LustreError: 161139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786343690 with bad export cookie 4169655660466544875 [ 3514.453820] Lustre: *** cfs_fail_loc=718, val=0*** [ 3514.455356] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3514.458039] LustreError: 161139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3514.459807] Lustre: Skipped 3 previous similar messages [ 3514.594636] Lustre: server umount lustre-MDT0001 complete [ 3518.445723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3524.064067] LustreError: 147611:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c6e4842d x1873116940816640/t0(0) o250->MGC192.168.203.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3524.183317] Lustre: *** cfs_fail_loc=723, val=0*** [ 3524.185029] Lustre: Skipped 1 previous similar message [ 3524.232069] LustreError: 164487:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-122) [ 3524.235664] LustreError: 164487:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.128@tcp: cfg command failed: rc = -122 [ 3524.239904] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3524.239904] [ 3524.244521] LustreError: MGC192.168.203.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -122. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3524.250723] LustreError: 164458:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -122 [ 3524.255248] LustreError: 164458:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -122 [ 3524.258474] LustreError: 164458:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3524.261515] LustreError: 164458:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3524.307987] Lustre: server umount lustre-MDT0000 complete [ 3524.310091] LustreError: 164458:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -122 [ 3526.939649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3526.980456] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3527.090800] LustreError: 162944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3527.091173] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3527.099166] LustreError: 162944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3527.103667] Lustre: Skipped 2 previous similar messages [ 3527.127526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3527.129991] Lustre: Skipped 2 previous similar messages [ 3527.150739] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3528.596568] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3531.624888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3531.724808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 3531.786522] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 3533.310895] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3534.955789] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3535.263141] Lustre: 147612:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786343695/real 1786343695] req@00000000523c36f7 x1873116940816256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786343711 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3535.794737] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3535.840075] Lustre: 147613:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786343696/real 1786343696] req@000000005da1a16f x1873116940816384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786343712 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3536.866487] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3536.870083] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3537.895715] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3537.895831] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3538.917295] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3539.935158] Lustre: 147614:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786343700/real 1786343700] req@0000000082e42bb5 x1873116940816768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786343716 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3539.946045] Lustre: 147614:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3540.085531] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3541.407968] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 3543.304329] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3543.877492] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3545.107857] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3550.178392] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3550.181215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3550.186100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3550.188788] Lustre: Skipped 5 previous similar messages [ 3554.509027] Lustre: server umount lustre-OST0000 complete [ 3568.607938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3568.610220] Lustre: Skipped 8 previous similar messages [ 3570.143095] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3570.204524] Lustre: server umount lustre-MDT0000 complete [ 3571.419144] LustreError: 161139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786343747 with bad export cookie 4169655660466545778 [ 3571.421893] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3571.422840] LustreError: 161139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3571.539441] Lustre: server umount lustre-MDT0001 complete [ 3575.070374] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 3576.136441] Key type lgssc unregistered [ 3576.263425] LNet: 167960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3576.266333] LNetError: 167960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3576.273394] LNet: Removed LNI 192.168.203.128@tcp [ 3576.609121] Key type .llcrypt unregistered [ 3576.610136] Key type ._llcrypt unregistered [ 3587.913420] Key type ._llcrypt registered [ 3587.914625] Key type .llcrypt registered [ 3587.949517] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3588.257239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3588.289670] alg: No test for adler32 (adler32-zlib) [ 3589.138706] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 3589.223860] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 3590.799147] Key type lgssc registered [ 3591.157319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3593.791657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3593.799192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3594.978724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3596.479699] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3599.595752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3599.762168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3601.304155] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3603.961305] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3605.905334] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3608.189936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3608.310079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3610.350847] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3612.858075] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3620.581562] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3626.003195] Lustre: *** cfs_fail_loc=718, val=0*** [ 3626.180257] Lustre: server umount lustre-MDT0000 complete [ 3627.505186] LustreError: 169785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786343803 with bad export cookie 17616875395546851405 [ 3627.507878] Lustre: *** cfs_fail_loc=718, val=0*** [ 3627.509127] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3627.509850] LustreError: 169785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3627.511671] Lustre: Skipped 3 previous similar messages [ 3627.624324] Lustre: server umount lustre-MDT0001 complete [ 3631.594762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3636.703628] LustreError: 169114:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a1e651ae x1873117199750528/t0(0) o250->MGC192.168.203.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3636.809767] Lustre: *** cfs_fail_loc=727, val=0*** [ 3636.854392] LustreError: 173222:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-57) [ 3636.857308] LustreError: 173222:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.128@tcp: cfg command failed: rc = -57 [ 3636.861287] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3636.861287] [ 3636.865573] LustreError: MGC192.168.203.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -57. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3636.871367] LustreError: 173193:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -57 [ 3636.875848] LustreError: 173193:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -57 [ 3636.880342] LustreError: 173193:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3636.882686] LustreError: 173193:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3636.921603] Lustre: server umount lustre-MDT0000 complete [ 3636.923905] LustreError: 173193:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -57 [ 3640.188159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3640.233522] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3640.343804] LustreError: 171630:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3640.343979] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3640.351466] LustreError: 171630:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3640.357762] Lustre: Skipped 1 previous similar message [ 3640.377812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3640.402392] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3641.794982] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3644.014428] LustreError: 173745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3644.021490] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3644.750120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3644.855046] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3644.858235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 3644.892623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3644.913664] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 3646.254060] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3646.943108] Lustre: 169115:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786343807/real 1786343807] req@0000000078b1a233 x1873117199750144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786343823 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3646.952598] Lustre: 169115:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3647.904112] Lustre: 169117:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786343808/real 1786343808] req@00000000270bc317 x1873117199750400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786343824 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3648.641524] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3649.131251] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3650.019735] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3650.663104] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 3650.703539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3651.171060] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3652.703093] Lustre: 169116:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786343813/real 1786343813] req@000000005c35e368 x1873117199750784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786343829 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3652.714479] Lustre: 169116:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 3652.919945] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3653.408734] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3654.475913] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3658.720924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3658.723240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3658.727526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3660.257639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3660.258350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3663.517153] Lustre: server umount lustre-OST0000 complete [ 3665.376505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3665.377315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3665.380430] Lustre: Skipped 1 previous similar message [ 3665.382283] Lustre: Skipped 1 previous similar message [ 3670.496106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3670.498077] Lustre: Skipped 2 previous similar messages [ 3675.615934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3675.617457] Lustre: Skipped 1 previous similar message [ 3679.199089] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3679.252851] Lustre: server umount lustre-MDT0000 complete [ 3679.712183] LustreError: 173728:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3679.717469] LustreError: 173728:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3680.422113] LustreError: 172649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786343856 with bad export cookie 17616875395546852581 [ 3680.424707] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3680.426197] LustreError: 172649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3680.537769] Lustre: server umount lustre-MDT0001 complete [ 3683.986901] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 3684.962831] Key type lgssc unregistered [ 3685.081324] LNet: 176791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3685.084076] LNetError: 176791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3685.092317] LNet: Removed LNI 192.168.203.128@tcp [ 3685.384937] Key type .llcrypt unregistered [ 3685.386215] Key type ._llcrypt unregistered [ 3692.933279] Key type ._llcrypt registered [ 3692.934449] Key type .llcrypt registered [ 3692.969649] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3693.250087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3693.261671] alg: No test for adler32 (adler32-zlib) [ 3694.110747] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 3694.203315] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 3695.783145] Key type lgssc registered [ 3696.132355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3698.788906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3698.796591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3699.947299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3701.310173] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3704.163434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3704.295598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3705.677538] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3708.077449] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3709.983530] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3712.386055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3712.514195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3714.621530] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3717.008125] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3724.835882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3729.891422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3729.891526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3729.895629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3729.911188] Lustre: Skipped 1 previous similar message [ 3731.750647] Lustre: server umount lustre-MDT0000 complete [ 3732.599565] LustreError: 178080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3732.606304] LustreError: 178080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3733.064124] LustreError: 178064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786343909 with bad export cookie 15670873035911628494 [ 3733.067875] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3733.068216] LustreError: 178064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3735.008926] LustreError: 178078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3735.009484] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3735.022206] Lustre: Skipped 1 previous similar message [ 3735.025488] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3735.028771] Lustre: Skipped 2 previous similar messages [ 3737.708270] LustreError: 178080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3737.719241] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.28@tcp (stopping) [ 3739.247590] Lustre: server umount lustre-MDT0001 complete [ 3741.531312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3745.916674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3746.000722] LustreError: 181940:0:(llog.c:1655:llog_backup()) MGC192.168.203.128@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 3746.003759] Lustre: 181940:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.128@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 3758.187269] LustreError: 181945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3758.192743] LustreError: 181945:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3762.283278] LustreError: 181946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3762.289990] LustreError: 181946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3772.895792] LustreError: 181963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3772.900530] LustreError: 181963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3788.907274] LustreError: 181965:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3788.914036] LustreError: 181965:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 3806.815394] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3806.818415] Lustre: 181940:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.128@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3823.583853] LustreError: 181963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3823.588750] LustreError: 181963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 21 previous similar messages [ 3847.653523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3849.100691] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3851.873765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3852.771467] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3852.773452] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd97a1fff91c2a0d6 [ 3852.859536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3852.887316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:193) [ 3854.142756] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3856.402145] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3856.899388] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3857.893509] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3857.895586] Lustre: Skipped 1 previous similar message [ 3858.504930] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3859.992874] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 1 sec [ 3866.323161] Lustre: server umount lustre-MDT0000 complete [ 3867.470080] LustreError: 178065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344043 with bad export cookie 15670873035911635158 [ 3867.474170] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3867.474343] LustreError: 178065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3867.581045] Lustre: server umount lustre-MDT0001 complete [ 3889.300962] Lustre: server umount lustre-OST0000 complete [ 3895.810306] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 3899.094354] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3910.529792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3910.627861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3910.642136] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3910.674082] Lustre: lustre-MDT0000: new disk, initializing [ 3910.693469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3910.698865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3912.012811] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3916.093727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3916.133509] Lustre: 187017:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3916.154969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3916.156635] Lustre: Skipped 1 previous similar message [ 3916.191724] Lustre: lustre-MDT0001: new disk, initializing [ 3916.230162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3916.233789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3917.465331] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3920.895423] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3922.607779] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3924.722875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3924.836082] Lustre: lustre-OST0000: new disk, initializing [ 3924.837793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3924.840720] Lustre: 188983:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3926.643187] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3928.245650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3928.249710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3928.282091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3930.165046] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3931.439746] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3931.504326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3932.724263] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3932.786463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3933.664440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3933.664838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3933.667661] LustreError: Skipped 1 previous similar message [ 3933.672885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3933.676041] Lustre: Skipped 2 previous similar messages [ 3938.787231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3938.789412] Lustre: Skipped 1 previous similar message [ 3939.471197] Lustre: server umount lustre-OST0000 complete [ 3941.855787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3941.858738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3941.863152] Lustre: Skipped 1 previous similar message [ 3946.775822] Lustre: server umount lustre-MDT0000 complete [ 3947.942227] LustreError: 187009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344124 with bad export cookie 15670873035911652182 [ 3947.945208] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3947.946935] LustreError: 187009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3948.056857] Lustre: server umount lustre-MDT0001 complete [ 3950.722359] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 02:42:06 (1786344126) [ 3952.766849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3952.917143] LustreError: 191392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3952.922423] LustreError: 191392:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [ 3952.938320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3952.941856] Lustre: Skipped 2 previous similar messages [ 3954.250174] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3956.919692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3958.385349] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3960.719283] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3962.439850] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3964.547066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3966.492711] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3968.925648] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3983.556525] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 02:42:38 (1786344158) [ 3987.423910] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3987.427053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3987.431013] Lustre: Skipped 1 previous similar message [ 3987.433600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3987.435204] Lustre: Skipped 3 previous similar messages [ 3990.605992] Lustre: server umount lustre-OST0000 complete [ 3993.055843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3993.059637] LustreError: Skipped 1 previous similar message [ 4003.808486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4003.811121] Lustre: Skipped 7 previous similar messages [ 4006.367105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4006.445605] Lustre: server umount lustre-MDT0000 complete [ 4007.684059] LustreError: 191373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344184 with bad export cookie 15670873035911653050 [ 4007.686784] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4007.688536] LustreError: 191373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4007.809178] Lustre: server umount lustre-MDT0001 complete [ 4011.234371] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 4012.171677] Key type lgssc unregistered [ 4012.288318] LNet: 195142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4012.290666] LNetError: 195142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4012.298244] LNet: Removed LNI 192.168.203.128@tcp [ 4012.580117] Key type .llcrypt unregistered [ 4012.581183] Key type ._llcrypt unregistered [ 4019.662854] Key type ._llcrypt registered [ 4019.664291] Key type .llcrypt registered [ 4019.705927] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 4020.219794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4020.231236] alg: No test for adler32 (adler32-zlib) [ 4021.084212] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 4021.172982] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 4022.751126] Key type lgssc registered [ 4023.067141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4025.423096] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4025.428627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4040.863434] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4045.983375] LustreError: 196362:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.128@tcp: failed processing log, type 4: rc = -110 [ 4051.103543] LustreError: 196362:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4051.108525] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4051.110841] LustreError: MGC192.168.203.128@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4051.114716] LustreError: 196362:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4051.118271] LustreError: 196362:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 4051.121262] LustreError: 196362:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4051.142953] Lustre: server umount lustre-OST0000 complete [ 4051.144244] LustreError: 196362:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4053.994583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4055.140837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4056.465752] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4059.348932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4059.466410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4060.794587] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4063.119316] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4064.860412] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4067.080306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4067.164416] LustreError: 198675:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4067.168730] Lustre: 198675:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.128@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4067.210682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4069.055615] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4071.366025] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4072.066547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4072.068758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4072.069767] Lustre: Skipped 2 previous similar messages [ 4072.072090] Lustre: Skipped 1 previous similar message [ 4077.536476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4077.539213] Lustre: Skipped 2 previous similar messages [ 4078.229329] Lustre: server umount lustre-MDT0000 complete [ 4079.464379] LustreError: 196841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344255 with bad export cookie 6431060805047528203 [ 4079.467041] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4079.469527] LustreError: 196841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4079.603042] Lustre: server umount lustre-MDT0001 complete [ 4091.223674] Lustre: server umount lustre-OST0000 complete [ 4094.393956] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 4097.448953] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 4108.131056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4108.213322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4108.224402] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4108.260584] Lustre: lustre-MDT0000: new disk, initializing [ 4108.280575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4108.286389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4109.583845] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4113.617309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4113.661583] Lustre: 202190:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4113.685991] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4113.687826] Lustre: Skipped 1 previous similar message [ 4113.718355] Lustre: lustre-MDT0001: new disk, initializing [ 4113.754735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4113.766668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4113.771556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4115.078554] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4118.700132] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4120.369946] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4122.492214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4122.583101] Lustre: lustre-OST0000: new disk, initializing [ 4122.585834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4122.589492] Lustre: 204161:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4122.615799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4123.259398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4123.262429] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4123.296900] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4124.516680] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4127.997871] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4129.293101] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4129.359887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4130.577888] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4130.641034] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4133.856663] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4133.856946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4133.858906] LustreError: Skipped 1 previous similar message [ 4133.859444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4133.862962] Lustre: Skipped 1 previous similar message [ 4133.866622] Lustre: Skipped 1 previous similar message [ 4137.291734] Lustre: server umount lustre-OST0000 complete [ 4138.976209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4138.976665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4138.979431] Lustre: Skipped 1 previous similar message [ 4138.982474] Lustre: Skipped 1 previous similar message [ 4144.096016] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4144.098790] Lustre: Skipped 1 previous similar message [ 4144.533911] Lustre: server umount lustre-MDT0000 complete [ 4145.676253] LustreError: 202985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344322 with bad export cookie 6431060805047528889 [ 4145.680172] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4145.681340] LustreError: 202985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4145.790295] Lustre: server umount lustre-MDT0001 complete [ 4148.340802] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 02:45:23 (1786344323) [ 4153.546573] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 4154.449345] Key type lgssc unregistered [ 4154.567290] LNet: 207103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4154.569420] LNetError: 207103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4154.576238] LNet: Removed LNI 192.168.203.128@tcp [ 4154.858823] Key type .llcrypt unregistered [ 4154.860301] Key type ._llcrypt unregistered [ 4161.456158] Key type ._llcrypt registered [ 4161.457359] Key type .llcrypt registered [ 4161.489864] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 4161.862529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4161.870797] alg: No test for adler32 (adler32-zlib) [ 4162.719689] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 4162.801253] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 4164.375136] Key type lgssc registered [ 4164.692492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4167.098161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4167.105052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4168.250103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4169.530147] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4172.184497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4172.292830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4173.602100] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4175.833838] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4177.540675] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4179.891333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4179.956771] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4179.958223] LustreError: 210164:0:(llog.c:1696:llog_backup()) MGC192.168.203.128@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4179.961844] Lustre: 210164:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.128@tcp: can't backup local config lustre-OST0000: rc = -28 [ 4179.995121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4181.756717] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4184.036849] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4187.104666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4187.106806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4187.108885] Lustre: Skipped 2 previous similar messages [ 4187.110715] Lustre: Skipped 1 previous similar message [ 4190.868580] Lustre: server umount lustre-MDT0000 complete [ 4192.002162] LustreError: 208324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344368 with bad export cookie 15884178702022423836 [ 4192.005024] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4192.006369] LustreError: 208324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4192.112572] Lustre: server umount lustre-MDT0001 complete [ 4203.478494] Lustre: server umount lustre-OST0000 complete [ 4206.311813] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 4209.421859] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 4220.122727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4220.199468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4220.208525] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4220.241509] Lustre: lustre-MDT0000: new disk, initializing [ 4220.259924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4220.266501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4221.407993] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4225.524486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4225.566127] Lustre: 213677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4225.583314] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4225.584459] Lustre: Skipped 1 previous similar message [ 4225.585636] LustreError: 214462:0:(llog.c:1696:llog_backup()) MGC192.168.203.128@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 4225.587842] LustreError: 214462:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4225.589379] Lustre: 214462:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.128@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 4225.596061] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4225.597505] Lustre: Skipped 1 previous similar message [ 4225.633268] Lustre: lustre-MDT0001: new disk, initializing [ 4225.666637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4225.678122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4225.681519] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4227.045890] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4230.853882] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4232.514501] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4234.570935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4234.655618] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4234.657234] Lustre: Skipped 1 previous similar message [ 4234.658750] LustreError: 215621:0:(llog.c:1696:llog_backup()) MGC192.168.203.128@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4234.662209] LustreError: 215621:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4234.664799] Lustre: 215621:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.128@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 4234.669065] Lustre: 215621:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 4234.677722] Lustre: lustre-OST0000: new disk, initializing [ 4234.680123] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4234.683899] Lustre: 215645:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4234.713560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4236.380561] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4239.974310] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4240.379345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4240.383038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4240.421732] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4241.408535] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4241.480271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4242.834235] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4242.904283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4245.476028] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4245.476722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4245.477925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4245.477930] Lustre: Skipped 1 previous similar message [ 4245.479272] LustreError: Skipped 1 previous similar message [ 4245.484094] Lustre: Skipped 1 previous similar message [ 4249.614850] Lustre: server umount lustre-OST0000 complete [ 4251.615880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4251.618682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4251.622704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4251.624477] Lustre: Skipped 1 previous similar message [ 4255.712558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4255.713377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4257.053225] Lustre: server umount lustre-MDT0000 complete [ 4258.241156] LustreError: 213671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344434 with bad export cookie 15884178702022424536 [ 4258.244941] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4258.246394] LustreError: 213671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4258.368288] Lustre: server umount lustre-MDT0001 complete [ 4261.162271] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 02:47:16 (1786344436) [ 4266.308599] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 4269.606960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4269.728243] LustreError: 218764:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4269.735911] LustreError: 218764:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4269.739808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4271.074494] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4273.863695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4275.303546] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4276.262613] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4278.418905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4280.532255] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4283.553967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4283.627796] Lustre: lustre-OST0001: new disk, initializing [ 4283.629839] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4283.632186] Lustre: 221069:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4285.652641] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4285.829353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4285.834935] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4285.873191] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4291.210602] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4294.860919] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4295.502476] Lustre: DEBUG MARKER: ADD OST3 [ 4297.230289] loop: module loaded [ 4297.248408] loop0: detected capacity change from 0 to 204800000 [ 4297.312606] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4297.318247] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4297.324681] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4299.360718] loop0: detected capacity change from 0 to 204800000 [ 4301.245526] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4301.325288] Lustre: lustre-OST0003: new disk, initializing [ 4301.327066] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 4301.329529] Lustre: 222799:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4302.313924] Lustre: DEBUG MARKER: STOP OST3 [ 4302.795601] Lustre: *** cfs_fail_loc=2109, val=2*** [ 4302.831419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 4302.835408] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 4302.846675] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 4302.987781] Lustre: Failing over lustre-OST0003 [ 4303.027686] Lustre: server umount lustre-OST0003 complete [ 4303.438401] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4304.800092] LustreError: 223125:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 4307.363367] Lustre: DEBUG MARKER: START OST3 again [ 4307.937405] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4307.940212] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4307.943791] LustreError: 224083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4307.944173] Lustre: Skipped 1 previous similar message [ 4309.508599] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4309.605785] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 4309.609413] Lustre: Skipped 4 previous similar messages [ 4309.615419] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 4311.010227] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4311.092112] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4311.092320] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4311.427225] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4324.923805] Lustre: Failing over lustre-OST0003 [ 4324.962764] Lustre: server umount lustre-OST0003 complete [ 4325.356773] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4325.358193] LustreError: 220248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4325.363338] Lustre: Skipped 1 previous similar message [ 4325.371617] LustreError: 220248:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4330.469243] LustreError: 220249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4330.477162] LustreError: 220249:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4335.589953] LustreError: 224083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4335.596069] LustreError: 224083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4340.705406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4340.707794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4340.710509] Lustre: Skipped 4 previous similar messages [ 4343.572075] Lustre: server umount lustre-MDT0000 complete [ 4344.703079] LustreError: 218745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344521 with bad export cookie 15884178702022425404 [ 4344.706519] LustreError: 218745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4344.706573] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4345.824072] LustreError: 218760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4345.824665] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4345.830788] LustreError: 218760:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4345.834622] Lustre: Skipped 4 previous similar messages [ 4349.856099] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4349.857825] Lustre: Skipped 3 previous similar messages [ 4359.135111] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4359.225538] Lustre: server umount lustre-MDT0001 complete [ 4360.576569] Lustre: server umount lustre-OST0000 complete [ 4361.881380] Lustre: server umount lustre-OST0001 complete [ 4364.170646] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 4366.680853] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 4377.961315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4378.034883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4378.044193] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4378.077746] Lustre: lustre-MDT0000: new disk, initializing [ 4378.096760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4378.103028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4379.389054] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4383.405053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4383.442768] Lustre: 228210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4383.551459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4384.808192] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4388.394252] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4390.011482] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4392.032021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4392.122015] Lustre: 230177:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4393.782985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4393.815456] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4393.867510] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4397.510863] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4398.950383] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4399.022818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4400.371316] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4400.438561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4404.192818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4404.194861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4404.195295] LustreError: Skipped 1 previous similar message [ 4404.198551] Lustre: Skipped 1 previous similar message [ 4404.199532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4404.204368] Lustre: Skipped 3 previous similar messages [ 4407.179171] Lustre: server umount lustre-OST0000 complete [ 4414.425347] Lustre: server umount lustre-MDT0000 complete [ 4414.432784] LustreError: 228221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4414.437527] LustreError: 228221:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4415.661277] LustreError: 230176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344592 with bad export cookie 15884178702022427658 [ 4415.665199] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4415.665452] LustreError: 230176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4415.789976] Lustre: server umount lustre-MDT0001 complete [ 4416.944913] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 02:49:52 (1786344592) [ 4421.941212] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 4424.917219] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 4436.519430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4436.601178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4436.610730] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4436.612542] Lustre: Skipped 3 previous similar messages [ 4436.648766] Lustre: lustre-MDT0000: new disk, initializing [ 4436.650175] Lustre: Skipped 2 previous similar messages [ 4436.674895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4436.677346] Lustre: Skipped 2 previous similar messages [ 4438.001716] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4441.998075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4442.042491] Lustre: 234779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4442.145647] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4443.479208] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4447.205499] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4449.056618] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4451.327092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4451.429036] Lustre: 236748:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4453.149655] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4453.335821] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4457.167236] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4458.688730] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4458.763432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4460.203018] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4460.284962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4463.584540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4463.587216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4463.587221] LustreError: Skipped 2 previous similar messages [ 4463.589465] Lustre: Skipped 2 previous similar messages [ 4475.359105] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4475.408199] Lustre: server umount lustre-OST0000 complete [ 4477.919751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4477.924563] Lustre: Skipped 5 previous similar messages [ 4482.648466] Lustre: server umount lustre-MDT0000 complete [ 4483.876377] LustreError: 234773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344660 with bad export cookie 15884178702022428540 [ 4483.880229] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4483.881327] LustreError: 234773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4484.004348] Lustre: server umount lustre-MDT0001 complete [ 4486.906833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4487.052187] LustreError: 238962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4488.420383] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4491.203475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4492.644518] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4495.041555] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4496.841696] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4499.018534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4501.017808] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4503.332842] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4521.952503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4521.956614] LustreError: Skipped 1 previous similar message [ 4526.225040] Lustre: server umount lustre-OST0000 complete [ 4528.097496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4528.099457] Lustre: Skipped 12 previous similar messages [ 4533.594467] Lustre: server umount lustre-MDT0000 complete [ 4534.875330] LustreError: 238943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344711 with bad export cookie 15884178702022429422 [ 4534.879033] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4534.879506] LustreError: 238943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4535.004648] Lustre: server umount lustre-MDT0001 complete [ 4636.858647] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 4638.057237] Key type lgssc unregistered [ 4638.198415] LNet: 242650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4638.201921] LNetError: 242650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4638.210335] LNet: Removed LNI 192.168.203.128@tcp [ 4638.567145] Key type .llcrypt unregistered [ 4638.568882] Key type ._llcrypt unregistered [ 4648.685439] Key type ._llcrypt registered [ 4648.686475] Key type .llcrypt registered [ 4648.721505] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 4654.730096] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 4655.047471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4655.081577] alg: No test for adler32 (adler32-zlib) [ 4655.946242] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 4656.028842] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 4657.607168] Key type lgssc registered [ 4657.940700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4668.792536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4668.797436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4669.872802] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4669.882326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4669.914565] Lustre: lustre-MDT0000: new disk, initializing [ 4669.931554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4669.937287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4671.223867] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4675.095215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4675.132891] Lustre: 245971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4675.152826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4675.154587] Lustre: Skipped 1 previous similar message [ 4675.185280] Lustre: lustre-MDT0001: new disk, initializing [ 4675.216509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4675.230307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4675.234574] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4676.548496] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4679.977680] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4681.819569] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4684.036547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4684.131613] Lustre: lustre-OST0000: new disk, initializing [ 4684.134246] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4684.137349] Lustre: 247940:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4684.164971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4686.019750] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4686.529903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4686.533512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4686.570039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4689.649176] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4691.086412] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4691.161913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4692.536175] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4692.607798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4697.058176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4697.059285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4697.061243] LustreError: Skipped 1 previous similar message [ 4697.061992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4697.065087] Lustre: Skipped 1 previous similar message [ 4699.336286] Lustre: server umount lustre-OST0000 complete [ 4701.151745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4701.154225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4701.158562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4701.161112] Lustre: Skipped 1 previous similar message [ 4702.176302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4702.176592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4706.650911] Lustre: server umount lustre-MDT0000 complete [ 4707.891177] LustreError: 245965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344884 with bad export cookie 15764206493800597410 [ 4707.893760] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4707.895413] LustreError: 245965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4708.008546] Lustre: server umount lustre-MDT0001 complete [ 4713.916158] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 4717.280366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4717.405475] LustreError: 250864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4717.411980] LustreError: 250864:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4717.417436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4718.804432] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4721.724286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4721.875772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4723.340114] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4724.368439] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4726.572878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4728.549642] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4731.351416] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4731.416193] Lustre: lustre-OST0001: new disk, initializing [ 4731.418445] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4731.421768] Lustre: 253170:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4731.454872] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4731.457225] Lustre: Skipped 1 previous similar message [ 4733.529622] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4738.926913] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4739.068745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4739.071669] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4739.102237] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4742.562378] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4746.318677] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 02:55:21 (1786344921) [ 4747.743855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4747.746644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4747.750557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4747.752228] Lustre: Skipped 1 previous similar message [ 4753.112933] Lustre: server umount lustre-MDT0000 complete [ 4754.274051] LustreError: 250846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786344930 with bad export cookie 15764206493800598278 [ 4754.277582] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4754.278878] LustreError: 250846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4754.380616] Lustre: server umount lustre-MDT0001 complete [ 4764.800975] Lustre: server umount lustre-OST0000 complete [ 4776.217838] Lustre: server umount lustre-OST0001 complete [ 4778.220522] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 4781.341668] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 4793.154060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4793.233635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4793.244288] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4793.280242] Lustre: lustre-MDT0000: new disk, initializing [ 4793.301345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4793.307343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4794.640446] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4798.744838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4798.787544] Lustre: 257109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4798.892249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4800.237776] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4803.908668] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4805.666957] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4807.911745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4808.011041] Lustre: lustre-OST0000: new disk, initializing [ 4808.012254] Lustre: Skipped 1 previous similar message [ 4808.014424] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4808.016995] Lustre: Skipped 2 previous similar messages [ 4808.019593] Lustre: 259077:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4809.273626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4809.277565] Lustre: Skipped 1 previous similar message [ 4809.280045] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4809.316212] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4809.953379] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4813.667505] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4815.099037] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4815.171175] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4816.485875] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4816.559239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4819.427257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4819.427445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4819.428109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4819.428113] Lustre: Skipped 4 previous similar messages [ 4819.430357] LustreError: Skipped 1 previous similar message [ 4819.434354] Lustre: Skipped 4 previous similar messages [ 4823.315807] Lustre: server umount lustre-OST0000 complete [ 4824.544491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4829.664038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4829.666281] Lustre: Skipped 4 previous similar messages [ 4830.612179] Lustre: server umount lustre-MDT0000 complete [ 4831.756236] LustreError: 259076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786345008 with bad export cookie 15764206493800600406 [ 4831.757257] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4831.761962] LustreError: 259076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4831.869870] Lustre: server umount lustre-MDT0001 complete [ 4834.555955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4834.693492] LustreError: 261291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4834.713608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4834.717019] Lustre: Skipped 2 previous similar messages [ 4836.057060] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4838.817873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4840.314199] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4842.712645] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4844.463132] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4846.618557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4848.714528] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4851.166228] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4993.504495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4993.505063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4993.506975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4993.513585] Lustre: Skipped 3 previous similar messages [ 4999.502032] Lustre: server umount lustre-OST0000 complete [ 5015.007113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5015.060624] Lustre: server umount lustre-MDT0000 complete [ 5016.253383] LustreError: 261272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786345192 with bad export cookie 15764206493800601288 [ 5016.256185] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5016.257593] LustreError: 261272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5016.378690] Lustre: server umount lustre-MDT0001 complete [ 5115.424114] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 5116.457256] Key type lgssc unregistered [ 5116.584344] LNet: 264910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5116.587178] LNetError: 264910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5116.597332] LNet: Removed LNI 192.168.203.128@tcp [ 5116.905131] Key type .llcrypt unregistered [ 5116.906354] Key type ._llcrypt unregistered [ 5124.896209] Key type ._llcrypt registered [ 5124.897614] Key type .llcrypt registered [ 5124.934253] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 5131.057410] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 5131.556599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5131.569765] alg: No test for adler32 (adler32-zlib) [ 5132.425857] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 5132.514982] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 5134.095138] Key type lgssc registered [ 5134.484256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5148.612345] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 5151.915942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5151.921647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5153.001164] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5153.010898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5153.045379] Lustre: lustre-MDT0000: new disk, initializing [ 5153.066130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5153.071686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5154.471949] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5158.703438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5158.747382] Lustre: 268953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5158.777705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5158.780967] Lustre: Skipped 1 previous similar message [ 5158.819839] Lustre: lustre-MDT0001: new disk, initializing [ 5158.855385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5158.871662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5158.876212] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5160.309894] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5162.579482] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5164.675474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5164.780460] Lustre: lustre-OST0000: new disk, initializing [ 5164.782340] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5164.785115] Lustre: 270584:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5164.819235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5166.706660] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5169.655193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5169.658820] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5169.689348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5170.714404] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5170.778750] Lustre: lustre-OST0001: new disk, initializing [ 5170.781292] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5170.783789] Lustre: 271458:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5170.816920] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5172.648676] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5177.346279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5177.351501] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5177.380572] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5178.006275] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5181.493165] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5184.480868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5184.483240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5184.500743] Lustre: server umount lustre-MDT0000 complete [ 5185.857590] LustreError: 272225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786345362 with bad export cookie 15776506211935261798 [ 5185.858908] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5185.861891] LustreError: 272225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5187.552626] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5187.554089] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5187.557639] Lustre: Skipped 3 previous similar messages [ 5187.559237] Lustre: Skipped 1 previous similar message [ 5190.559967] LustreError: 268958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5190.560402] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5190.566037] LustreError: 268958:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5190.567874] Lustre: Skipped 1 previous similar message [ 5192.044397] Lustre: server umount lustre-MDT0001 complete [ 5193.270238] Lustre: server umount lustre-OST0000 complete [ 5206.541656] Lustre: server umount lustre-OST0001 complete [ 5211.050785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5214.080271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5216.882680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5220.022544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5226.303924] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 5230.042495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5230.053952] Lustre: lustre-MDT0000: reset Object Index mappings [ 5230.223991] LustreError: 276437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5230.244094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5231.659618] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5234.547830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5234.554812] Lustre: lustre-MDT0001: reset Object Index mappings [ 5236.103236] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5237.083298] Lustre: 277579:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5239.314134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5239.446822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5239.451186] Lustre: Skipped 1 previous similar message [ 5240.485819] LustreError: 277932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5240.491080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5241.444297] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5244.099112] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5245.227192] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 5245.964191] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5248.860319] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5249.955495] Lustre: 279445:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5255.135893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5255.137843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5255.141698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5257.181737] Lustre: server umount lustre-MDT0000 complete [ 5258.369187] LustreError: 276418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786345434 with bad export cookie 15776506211935268413 [ 5258.371600] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5258.373820] LustreError: 276418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5258.491957] Lustre: server umount lustre-MDT0001 complete [ 5270.017313] Lustre: server umount lustre-OST0000 complete [ 5281.503461] Lustre: server umount lustre-OST0001 complete [ 5287.734214] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:04:23 (1786345463) [ 5292.978443] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 5296.081341] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 5307.823017] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5307.905588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5307.915820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5307.951374] Lustre: lustre-MDT0000: new disk, initializing [ 5307.973611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5307.976450] Lustre: Skipped 1 previous similar message [ 5307.982976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5309.391562] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5313.589788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5313.630362] Lustre: 283689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5313.738341] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5315.046658] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5318.660676] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5320.475650] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5322.797388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5322.915368] Lustre: lustre-OST0000: new disk, initializing [ 5322.916920] Lustre: Skipped 1 previous similar message [ 5322.918427] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5322.920478] Lustre: Skipped 2 previous similar messages [ 5322.922451] Lustre: 285654:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5323.963211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5323.966826] Lustre: Skipped 1 previous similar message [ 5323.969636] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5323.998034] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5324.907850] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5328.684302] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5330.082040] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5330.152088] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5331.504660] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5331.584257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5334.500278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5334.501488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5334.502991] LustreError: Skipped 1 previous similar message [ 5334.507159] Lustre: Skipped 3 previous similar messages [ 5334.507838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5334.512237] Lustre: Skipped 4 previous similar messages [ 5338.322208] Lustre: server umount lustre-OST0000 complete [ 5339.616510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5339.616731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5339.617376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5339.624827] Lustre: Skipped 2 previous similar messages [ 5345.776623] Lustre: server umount lustre-MDT0000 complete [ 5347.057474] LustreError: 283681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786345523 with bad export cookie 15776506211935270492 [ 5347.060788] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5347.062070] LustreError: 283681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5347.179622] Lustre: server umount lustre-MDT0001 complete [ 5353.608867] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 5356.767240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5356.908150] LustreError: 288584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5356.923513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5356.925470] Lustre: Skipped 2 previous similar messages [ 5358.217227] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5361.004443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5362.440212] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5363.396789] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5365.588392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5367.727606] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5370.534699] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5370.595110] Lustre: lustre-OST0001: new disk, initializing [ 5370.596780] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5370.599204] Lustre: 290892:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5371.966154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5371.969877] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5372.006360] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5372.551199] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5376.973458] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5378.161994] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5379.920188] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 5379.943805] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 5382.114766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5382.116751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5382.118642] Lustre: Skipped 2 previous similar messages [ 5382.121808] Lustre: Skipped 6 previous similar messages [ 5387.743035] Lustre: server umount lustre-MDT0000 complete [ 5388.951227] LustreError: 288567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786345565 with bad export cookie 15776506211935271374 [ 5388.954658] LustreError: 288567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5388.955345] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5389.061823] Lustre: server umount lustre-MDT0001 complete [ 5400.580589] Lustre: server umount lustre-OST0000 complete [ 5412.132913] Lustre: server umount lustre-OST0001 complete [ 5417.297525] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 5420.660708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5420.797598] LustreError: 293937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5422.106069] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5424.864020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5424.985677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5424.988926] Lustre: Skipped 4 previous similar messages [ 5426.320285] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5427.322068] Lustre: 295077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5429.493841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5431.512557] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5431.652306] LustreError: 295431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5431.663576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 5434.574607] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5435.758857] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 5436.708727] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5439.757671] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5440.936684] Lustre: 296954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5442.765581] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 5442.771033] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 5445.878907] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:07:01 (1786345621) [ 5460.959945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5460.963346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5460.968539] Lustre: Skipped 1 previous similar message [ 5460.970769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5460.972707] Lustre: Skipped 4 previous similar messages [ 5463.464214] Lustre: server umount lustre-MDT0000 complete [ 5464.647293] LustreError: 293918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786345641 with bad export cookie 15776506211935277653 [ 5464.651506] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5464.652882] LustreError: 293918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5466.592281] LustreError: 293966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5466.598895] LustreError: 293966:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5470.841612] Lustre: server umount lustre-MDT0001 complete [ 5478.209660] Lustre: server umount lustre-OST0000 complete [ 5485.524892] Lustre: server umount lustre-OST0001 complete [ 5487.623748] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 5490.722562] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 5505.225553] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 5508.566977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5508.652298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5508.662052] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5508.700750] Lustre: lustre-MDT0000: new disk, initializing [ 5508.727263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5510.106454] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5515.675712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5515.803197] Lustre: 301183:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5516.010453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5519.038961] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5525.777148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5525.935326] Lustre: 302814:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5527.575139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5529.687441] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5536.416582] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5536.528523] Lustre: 303684:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5538.339107] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5540.252328] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5545.338852] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5546.818624] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5546.822078] Lustre: Skipped 1 previous similar message [ 5550.799938] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 03:08:46 (1786345726) [ 5552.095676] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5552.098273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5552.101322] Lustre: Skipped 5 previous similar messages [ 5552.102791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5552.104576] Lustre: Skipped 8 previous similar messages [ 5557.603266] Lustre: server umount lustre-MDT0000 complete [ 5558.754096] LustreError: 301194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5558.769515] LustreError: 301194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5559.394375] LustreError: 301177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786345735 with bad export cookie 15776506211935285535 [ 5559.400535] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5559.405416] LustreError: 301177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5559.606101] Lustre: server umount lustre-MDT0001 complete [ 5571.087968] Lustre: server umount lustre-OST0000 complete [ 5581.940040] Lustre: server umount lustre-OST0001 complete [ 5585.463955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5585.664474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5585.667539] Lustre: Skipped 6 previous similar messages [ 5587.128828] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5590.372038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5592.201431] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5594.876761] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5596.833069] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5599.473652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5601.907482] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5604.862586] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5617.013038] loop1: detected capacity change from 0 to 204800000 [ 5617.112526] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5617.117805] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5617.127219] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5619.171946] loop1: detected capacity change from 0 to 204800000 [ 5621.205167] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5621.237696] LustreError: 305916:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs23793: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 5621.243756] LustreError: 305916:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs23793 [ 5621.246374] LustreError: 305916:0:(mgs_handler.c:517:mgs_target_reg()) fs23793-MDT0001: Error checking target index: rc = -13 [ 5621.250704] LustreError: fs23793-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5621.254412] LustreError: 309643:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 5621.257600] LustreError: 309643:0:(tgt_mount.c:2129:server_put_super()) no obd fs23793-MDT0001 [ 5621.259586] LustreError: 309643:0:(tgt_mount.c:116:server_deregister_mount()) fs23793-MDT0001 not registered [ 5621.270604] Lustre: server umount fs23793-MDT0001 complete [ 5621.272196] LustreError: 309643:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5624.200441] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5624.235854] Lustre: Setting parameter fs23793-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs23793-MDT0001 [ 5624.257950] Lustre: srv-fs23793-MDT0001: No data found on store. Initialize space. [ 5624.260118] Lustre: Skipped 4 previous similar messages [ 5624.298815] Lustre: fs23793-MDT0001: new disk, initializing [ 5624.300585] Lustre: Skipped 3 previous similar messages [ 5624.336626] Lustre: srv-fs23793-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 5624.339780] Lustre: 310108:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs23793-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5624.343902] LustreError: 310108:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs23793-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5625.314594] LustreError: 307755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5625.321967] LustreError: 307755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 5625.810537] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5628.927487] Lustre: server umount fs23793-MDT0001 complete [ 5633.934090] loop1: detected capacity change from 0 to 204800000 [ 5636.035162] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5636.168500] Lustre: 311677:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs23793-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5636.172583] LustreError: 311677:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs23793-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5637.736996] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5639.592903] Lustre: server umount fs23793-MDT0001 complete [ 5645.795645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5645.796715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5645.801906] Lustre: Skipped 5 previous similar messages [ 5645.804553] Lustre: Skipped 5 previous similar messages [ 5649.144106] Lustre: server umount lustre-MDT0000 complete [ 5652.428407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5652.487925] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5654.192753] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5656.987384] loop1: detected capacity change from 0 to 204800000 [ 5658.085843] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5659.013244] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5659.124677] Lustre: 314118:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs23793-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5659.127548] LustreError: 314118:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs23793-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5660.646163] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5662.645592] Lustre: server umount fs23793-MDT0001 complete [ 5672.416803] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5676.692190] Lustre: server umount lustre-OST0000 complete [ 5684.387821] Lustre: server umount lustre-MDT0000 complete [ 5685.976466] LustreError: 310654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786345862 with bad export cookie 15776506211935288209 [ 5685.980417] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5685.990886] LustreError: 310654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5686.156224] Lustre: server umount lustre-MDT0001 complete [ 5690.989589] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 5692.370816] Key type lgssc unregistered [ 5692.537569] LNet: 316459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5692.541761] LNetError: 316459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5692.555610] LNet: Removed LNI 192.168.203.128@tcp [ 5693.021204] Key type .llcrypt unregistered [ 5693.022534] Key type ._llcrypt unregistered [ 5702.656454] Key type ._llcrypt registered [ 5702.658438] Key type .llcrypt registered [ 5702.713729] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 5703.217946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5703.228484] alg: No test for adler32 (adler32-zlib) [ 5704.121947] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 5704.249774] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 5705.855803] Key type lgssc registered [ 5706.462025] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5710.364890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5710.374446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5711.570726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5713.474039] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5716.931360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5717.149747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5718.834406] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5721.804459] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5723.957886] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5726.698862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5726.858914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5727.917240] LustreError: 319533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5727.925940] LustreError: 319533:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5729.332714] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5732.214652] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5733.350100] LustreError: 319534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5736.417747] LustreError: 319686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5736.423274] LustreError: 319686:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5741.538604] LustreError: 319534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5741.543914] LustreError: 319534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5743.327239] loop1: detected capacity change from 0 to 204800000 [ 5743.385589] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5743.389799] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5743.395770] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5745.115141] loop1: detected capacity change from 0 to 204800000 [ 5746.663624] LustreError: 319534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5746.670191] LustreError: 319534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5746.924031] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5746.970248] Lustre: fs22215-OST0000: new disk, initializing [ 5746.971984] Lustre: srv-fs22215-OST0000: No data found on store. Initialize space. [ 5746.974466] Lustre: 321440:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5746.996831] Lustre: fs22215-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5748.830403] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5752.202886] Lustre: server umount fs22215-OST0000 complete [ 5755.958876] loop1: detected capacity change from 0 to 204800000 [ 5756.899492] LustreError: 319686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5756.904896] LustreError: 319686:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5757.785400] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5757.841508] Lustre: fs22215-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5759.716518] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5762.468564] loop2: detected capacity change from 0 to 204800000 [ 5762.548562] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5762.554217] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5762.562319] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5764.212029] loop2: detected capacity change from 0 to 204800000 [ 5766.066956] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5766.095620] LustreError: fs22215-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 5766.101471] LustreError: fs22215-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5766.104442] LustreError: 324478:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 5766.106790] LustreError: 324478:0:(tgt_mount.c:2129:server_put_super()) no obd fs22215-OST0001 [ 5766.109511] LustreError: 324478:0:(tgt_mount.c:116:server_deregister_mount()) fs22215-OST0001 not registered [ 5766.119485] Lustre: server umount fs22215-OST0001 complete [ 5766.120949] LustreError: 324478:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5769.455314] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5769.500206] Lustre: fs22215-OST0001: new disk, initializing [ 5769.503723] Lustre: srv-fs22215-OST0001: No data found on store. Initialize space. [ 5769.506953] Lustre: 325027:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5769.531367] Lustre: fs22215-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5771.490059] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5773.420420] LustreError: 319534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5773.425647] LustreError: 319534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 5775.032278] Lustre: server umount fs22215-OST0000 complete [ 5778.183690] Lustre: server umount fs22215-OST0001 complete [ 5781.975659] loop1: detected capacity change from 0 to 204800000 [ 5783.834407] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5785.789209] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5787.858412] loop2: detected capacity change from 0 to 204800000 [ 5789.583705] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5789.642660] Lustre: fs22215-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5789.644746] Lustre: Skipped 1 previous similar message [ 5791.339392] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5794.042068] Lustre: server umount fs22215-OST0000 complete [ 5796.921083] Lustre: server umount fs22215-OST0001 complete [ 5802.977645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5802.979103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5802.980886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5802.984518] Lustre: Skipped 1 previous similar message [ 5802.988547] Lustre: Skipped 1 previous similar message [ 5806.480973] Lustre: server umount lustre-OST0000 complete [ 5808.096628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5808.097314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5808.100016] Lustre: Skipped 1 previous similar message [ 5813.216132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5813.219164] Lustre: Skipped 2 previous similar messages [ 5813.783307] Lustre: server umount lustre-MDT0000 complete [ 5814.932937] LustreError: 317691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786345991 with bad export cookie 10075394896542542558 [ 5814.936371] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5814.936566] LustreError: 317691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5815.048312] Lustre: server umount lustre-MDT0001 complete [ 5818.440095] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 5819.454791] Key type lgssc unregistered [ 5819.583428] LNet: 330940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5819.586353] LNetError: 330940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5819.593362] LNet: Removed LNI 192.168.203.128@tcp [ 5819.907148] Key type .llcrypt unregistered [ 5819.908790] Key type ._llcrypt unregistered [ 5829.904202] Key type ._llcrypt registered [ 5829.905954] Key type .llcrypt registered [ 5829.940726] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 5835.865079] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 5836.177502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5836.211774] alg: No test for adler32 (adler32-zlib) [ 5837.059178] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 5837.143517] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 5838.719128] Key type lgssc registered [ 5839.057573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5850.575024] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5850.581754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5851.665552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5851.675133] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5851.710173] Lustre: lustre-MDT0000: new disk, initializing [ 5851.730557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5851.736319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5853.019971] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5857.168884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5857.220812] Lustre: 334768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5857.252926] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5857.255089] Lustre: Skipped 1 previous similar message [ 5857.295433] Lustre: lustre-MDT0001: new disk, initializing [ 5857.329789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5857.343401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5857.348538] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5858.705141] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5862.539420] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5864.428475] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5866.774184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5866.874181] Lustre: lustre-OST0000: new disk, initializing [ 5866.876579] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5866.879391] Lustre: 336737:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5866.911086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5868.841923] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5872.570893] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5874.873047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5874.876813] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5874.909202] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5880.305637] Lustre: Mounted lustre-client - version 2.17.56_55_g704561f [ 5880.627847] Lustre: Unmounted lustre-client [ 5885.409941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5885.410055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5885.411038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5885.412788] LustreError: Skipped 1 previous similar message [ 5885.417062] Lustre: Skipped 1 previous similar message [ 5889.683591] Lustre: server umount lustre-OST0000 complete [ 5893.087867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5893.090940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5893.096284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5893.098585] Lustre: Skipped 1 previous similar message [ 5895.647992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5895.648392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5900.767857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5900.769776] Lustre: Skipped 2 previous similar messages [ 5904.864671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5904.866842] Lustre: Skipped 1 previous similar message [ 5905.375082] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5905.436190] Lustre: server umount lustre-MDT0000 complete [ 5906.544857] LustreError: 334761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786346082 with bad export cookie 6437143253097500768 [ 5906.547482] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5906.548742] LustreError: 334761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5906.653190] Lustre: server umount lustre-MDT0001 complete [ 5909.999845] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 5910.959359] Key type lgssc unregistered [ 5929.503420] LNet: 338900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5929.507314] LNetError: 338900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6002.213262] LNet: Removed LNI 192.168.203.128@tcp [ 6002.504202] Key type .llcrypt unregistered [ 6002.505148] Key type ._llcrypt unregistered [ 6007.843969] Key type ._llcrypt registered [ 6007.845158] Key type .llcrypt registered [ 6007.883851] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 6014.317749] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 6014.614849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6014.622252] alg: No test for adler32 (adler32-zlib) [ 6015.478411] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 6015.569207] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 6017.143115] Key type lgssc registered [ 6017.517490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6027.232119] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 03:16:42 (1786346202) [ 6029.396317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6029.401690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6030.485733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6030.498173] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6030.544908] Lustre: lustre-MDT0000: new disk, initializing [ 6030.568609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6030.575296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6031.784044] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6035.872977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6035.910737] Lustre: 342094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6035.934839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6035.936989] Lustre: Skipped 1 previous similar message [ 6035.976358] Lustre: lustre-MDT0001: new disk, initializing [ 6036.007387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6036.019683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6036.022962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6037.288557] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6040.770298] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6042.393512] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6044.331421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6044.423742] Lustre: lustre-OST0000: new disk, initializing [ 6044.426439] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6044.430111] Lustre: 344061:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6044.458565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6046.140197] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6046.900213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6046.903142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6046.928063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6049.545538] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6056.846058] Lustre: server umount lustre-OST0000 complete [ 6061.535744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6061.537947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6061.541838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6062.560423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6062.560560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6067.680660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6067.683152] Lustre: Skipped 2 previous similar messages [ 6067.707580] Lustre: server umount lustre-MDT0000 complete [ 6068.905165] LustreError: 342890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786346245 with bad export cookie 7558218206870229205 [ 6068.908349] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6068.908530] LustreError: 342890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6079.506811] Lustre: server umount lustre-MDT0001 complete [ 6082.845644] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6083.824592] Key type lgssc unregistered [ 6083.960451] LNet: 345972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6083.963378] LNetError: 345972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6083.971381] LNet: Removed LNI 192.168.203.128@tcp [ 6084.311914] Key type .llcrypt unregistered [ 6084.313257] Key type ._llcrypt unregistered [ 6091.897798] Key type ._llcrypt registered [ 6091.899398] Key type .llcrypt registered [ 6091.940220] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 6092.288823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6092.325085] alg: No test for adler32 (adler32-zlib) [ 6093.179200] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 6093.260100] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 6094.831194] Key type lgssc registered [ 6095.226296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6097.898256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6097.904467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6099.060080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6100.506192] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6103.459954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6103.586490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6105.013650] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6107.480792] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6109.237467] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6111.454371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6111.567597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6113.537280] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6115.880505] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6129.532818] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6130.061834] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6134.240788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6134.243823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6134.250173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6135.264976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6135.267938] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6136.436459] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.28@tcp (stopping) [ 6136.438685] Lustre: Skipped 2 previous similar messages [ 6137.624146] Lustre: server umount lustre-MDT0000 complete [ 6138.917344] LustreError: 347203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786346315 with bad export cookie 10554530085506868207 [ 6138.920511] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6138.920742] LustreError: 347203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6140.384091] LustreError: 347957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6140.384312] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6140.389173] LustreError: 347957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6140.400624] Lustre: Skipped 1 previous similar message [ 6140.403841] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6141.548280] LustreError: 347218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6143.391933] LustreError: 347216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6146.667625] LustreError: 349712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6146.667978] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.28@tcp (stopping) [ 6146.678288] Lustre: Skipped 2 previous similar messages [ 6151.787332] LustreError: 349712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6151.794880] LustreError: 349712:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6153.183097] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6153.270309] Lustre: server umount lustre-MDT0001 complete [ 6160.663782] Lustre: server umount lustre-OST0000 complete [ 6163.510640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6164.871500] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6165.783785] Lustre: 351420:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.128@tcp [ 6166.477160] Lustre: server umount MGS complete [ 6169.249642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6169.416801] LustreError: 351989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6169.442077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6170.790658] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6173.525844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6173.635466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6174.968834] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6177.813992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6179.843310] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6182.224904] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6182.722178] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 6184.469570] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6184.990647] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6189.536046] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6189.538207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6189.541898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6189.543431] Lustre: Skipped 3 previous similar messages [ 6193.634524] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6193.637304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6194.774602] Lustre: server umount lustre-OST0000 complete [ 6202.139137] Lustre: server umount lustre-MDT0000 complete [ 6203.347178] LustreError: 351972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786346379 with bad export cookie 10554530085506869033 [ 6203.350654] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6203.352580] LustreError: 351972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6203.463133] Lustre: server umount lustre-MDT0001 complete [ 6207.246432] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6208.257533] Key type lgssc unregistered [ 6208.387415] LNet: 355424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6208.389795] LNetError: 355424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6208.398264] LNet: Removed LNI 192.168.203.128@tcp [ 6208.701949] Key type .llcrypt unregistered [ 6208.703108] Key type ._llcrypt unregistered [ 6214.514086] Key type ._llcrypt registered [ 6214.515559] Key type .llcrypt registered [ 6214.553336] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 6220.774589] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 6221.240265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6221.254838] alg: No test for adler32 (adler32-zlib) [ 6222.105704] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 6222.190989] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 6223.767194] Key type lgssc registered [ 6224.119852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6235.171116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6235.176621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6236.257232] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6236.266961] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6236.301216] Lustre: lustre-MDT0000: new disk, initializing [ 6236.322691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6236.328979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6237.753526] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6241.932492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6241.975828] Lustre: 358555:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6242.000865] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6242.003216] Lustre: Skipped 1 previous similar message [ 6242.040228] Lustre: lustre-MDT0001: new disk, initializing [ 6242.074147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6242.087393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6242.091796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6243.423225] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6247.074834] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6248.780541] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6251.012521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6251.143626] Lustre: lustre-OST0000: new disk, initializing [ 6251.145775] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6251.148522] Lustre: 360523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6251.183381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6253.171815] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6256.907282] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6258.410338] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6260.729384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6260.732881] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6260.764306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6261.550571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6262.915383] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6262.980485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6265.828150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6265.829447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6265.830838] LustreError: Skipped 1 previous similar message [ 6265.835914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6265.838885] Lustre: Skipped 1 previous similar message [ 6269.711727] Lustre: server umount lustre-OST0000 complete [ 6270.945535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6270.946109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6270.949520] Lustre: Skipped 2 previous similar messages [ 6276.064196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6276.066306] Lustre: Skipped 2 previous similar messages [ 6281.184558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6281.186742] Lustre: Skipped 1 previous similar message [ 6285.279084] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6285.343375] Lustre: server umount lustre-MDT0000 complete [ 6286.604260] LustreError: 358548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786346462 with bad export cookie 9364446750446908447 [ 6286.607599] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6286.608193] LustreError: 358548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6286.739547] Lustre: server umount lustre-MDT0001 complete [ 6289.634300] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 03:21:04 (1786346464) [ 6294.635238] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 6297.744714] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 6309.304425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6309.391449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6309.403462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6309.440248] Lustre: lustre-MDT0000: new disk, initializing [ 6309.460967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6309.467495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6310.869755] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6315.052932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6315.094709] Lustre: 365292:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6315.119527] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6315.122979] Lustre: Skipped 1 previous similar message [ 6315.160490] Lustre: lustre-MDT0001: new disk, initializing [ 6315.190748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6315.202390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6315.205681] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6316.549463] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6320.167837] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6321.938368] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6324.105625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6324.201504] Lustre: lustre-OST0000: new disk, initializing [ 6324.203369] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6324.206045] Lustre: 367260:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6324.232709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6326.129637] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6326.774446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6326.778698] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6326.814097] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6329.807429] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6342.113161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6342.113818] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6342.115661] LustreError: Skipped 1 previous similar message [ 6342.116353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6342.120641] Lustre: Skipped 1 previous similar message [ 6342.126373] Lustre: Skipped 1 previous similar message [ 6345.170658] Lustre: server umount lustre-OST0000 complete [ 6347.232773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6347.236731] Lustre: Skipped 1 previous similar message [ 6352.352039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6352.354382] Lustre: Skipped 3 previous similar messages [ 6352.540746] Lustre: server umount lustre-MDT0000 complete [ 6353.738000] LustreError: 365286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786346530 with bad export cookie 9364446750446909315 [ 6353.741595] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6353.743022] LustreError: 365286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6353.856406] Lustre: server umount lustre-MDT0001 complete [ 6357.623588] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6358.614766] Key type lgssc unregistered [ 6358.753376] LNet: 369412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6358.756500] LNetError: 369412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6358.764304] LNet: Removed LNI 192.168.203.128@tcp [ 6359.074385] Key type .llcrypt unregistered [ 6359.076052] Key type ._llcrypt unregistered [ 6390.845971] Key type ._llcrypt registered [ 6390.847194] Key type .llcrypt registered [ 6390.888404] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6391.006221] Key type .llcrypt unregistered [ 6391.007669] Key type ._llcrypt unregistered [ 6418.293937] Key type ._llcrypt registered [ 6418.295093] Key type .llcrypt registered [ 6418.332592] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6418.447349] Key type .llcrypt unregistered [ 6418.448408] Key type ._llcrypt unregistered [ 6429.642684] Key type ._llcrypt registered [ 6429.644127] Key type .llcrypt registered [ 6429.690764] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6429.816279] Key type .llcrypt unregistered [ 6429.817409] Key type ._llcrypt unregistered [ 6440.848054] Key type ._llcrypt registered [ 6440.849261] Key type .llcrypt registered [ 6440.895202] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6441.026236] Key type .llcrypt unregistered [ 6441.027801] Key type ._llcrypt unregistered [ 6468.209524] Key type ._llcrypt registered [ 6468.210654] Key type .llcrypt registered [ 6468.247835] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6468.363186] Key type .llcrypt unregistered [ 6468.364285] Key type ._llcrypt unregistered [ 6479.262086] Key type ._llcrypt registered [ 6479.263195] Key type .llcrypt registered [ 6479.301213] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6479.405411] Key type .llcrypt unregistered [ 6479.407087] Key type ._llcrypt unregistered [ 6490.629603] Key type ._llcrypt registered [ 6490.630672] Key type .llcrypt registered [ 6490.668944] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6490.777231] Key type .llcrypt unregistered [ 6490.778667] Key type ._llcrypt unregistered [ 6501.503739] Key type ._llcrypt registered [ 6501.505627] Key type .llcrypt registered [ 6501.540908] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6501.652230] Key type .llcrypt unregistered [ 6501.653302] Key type ._llcrypt unregistered [ 6512.830922] Key type ._llcrypt registered [ 6512.832161] Key type .llcrypt registered [ 6512.868307] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6512.994329] Key type .llcrypt unregistered [ 6512.996155] Key type ._llcrypt unregistered [ 6524.060960] Key type ._llcrypt registered [ 6524.061976] Key type .llcrypt registered [ 6524.099116] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6524.201436] Key type .llcrypt unregistered [ 6524.202621] Key type ._llcrypt unregistered [ 6545.313643] Key type ._llcrypt registered [ 6545.314792] Key type .llcrypt registered [ 6545.351943] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6545.458232] Key type .llcrypt unregistered [ 6545.459456] Key type ._llcrypt unregistered [ 6556.494170] Key type ._llcrypt registered [ 6556.495448] Key type .llcrypt registered [ 6556.545357] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6556.684301] Key type .llcrypt unregistered [ 6556.685707] Key type ._llcrypt unregistered [ 6567.665431] Key type ._llcrypt registered [ 6567.666642] Key type .llcrypt registered [ 6567.706829] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6567.815338] Key type .llcrypt unregistered [ 6567.816428] Key type ._llcrypt unregistered [ 6585.796041] Key type ._llcrypt registered [ 6585.797663] Key type .llcrypt registered [ 6585.833283] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 6591.953206] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 6592.321587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6592.352495] alg: No test for adler32 (adler32-zlib) [ 6593.221935] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 6593.310428] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 6594.887130] Key type lgssc registered [ 6595.244597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6606.253286] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6606.261297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6607.348636] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 6607.358444] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 6607.393406] Lustre: tst250fs-MDT0000: new disk, initializing [ 6607.412987] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6607.418826] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6608.809905] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6612.853544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6612.891962] Lustre: 389364:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6612.914905] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 6612.917042] Lustre: Skipped 1 previous similar message [ 6612.953308] Lustre: tst250fs-MDT0001: new disk, initializing [ 6612.988697] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6613.000593] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6613.005166] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6614.331113] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6616.577737] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6618.695788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6618.798807] Lustre: tst250fs-OST0000: new disk, initializing [ 6618.800639] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 6618.803163] Lustre: 390995:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6618.833553] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6620.717463] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6624.779692] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6624.843911] Lustre: tst250fs-OST0001: new disk, initializing [ 6624.845642] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 6624.848794] Lustre: 391857:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6624.888428] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6625.257389] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6625.262659] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6625.350877] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6625.367429] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6626.936103] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6629.417687] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 6640.609601] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6640.612684] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 6640.613934] Lustre: Skipped 3 previous similar messages [ 6640.616041] Lustre: Skipped 2 previous similar messages [ 6645.729949] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 6645.732727] Lustre: Skipped 4 previous similar messages [ 6645.796553] Lustre: server umount tst250fs-MDT0000 complete [ 6646.969642] LustreError: 392478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786346823 with bad export cookie 11088867700713863850 [ 6646.973735] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6646.975832] LustreError: 392478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6647.105707] Lustre: server umount tst250fs-MDT0001 complete [ 6658.058203] Lustre: server umount tst250fs-OST0000 complete [ 6669.527628] Lustre: server umount tst250fs-OST0001 complete [ 6675.320488] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 6677.895047] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 6687.699137] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 03:27:43 (1786346863) [ 6688.205094] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 6693.229051] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 6696.357046] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 6706.542543] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6450 sec ========== 03:28:01 (1786346881) [ 6707.068905] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:28:02 (1786346882) === [ 6708.156099] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:28:03 (1786346883) === [ 6721.038154] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6722.023099] Key type lgssc unregistered [ 6722.147365] LNet: 400976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6722.149815] LNetError: 400976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6722.156277] LNet: Removed LNI 192.168.203.128@tcp [ 6722.446101] Key type .llcrypt unregistered [ 6722.447393] Key type ._llcrypt unregistered