[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 487810968 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2480MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003278] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.007891] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009019] pid_max: default: 32768 minimum: 301 [ 0.010161] LSM: Security Framework initializing [ 0.011073] Yama: becoming mindful. [ 0.012052] SELinux: Initializing. [ 0.013094] *** VALIDATE selinux *** [ 0.022193] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027297] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028173] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029131] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030125] *** VALIDATE tmpfs *** [ 0.032538] *** VALIDATE proc *** [ 0.034197] *** VALIDATE cgroup *** [ 0.035014] *** VALIDATE cgroup2 *** [ 0.037280] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040035] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.044053] debug: unmapping init [mem 0xffffffffbb259000-0xffffffffbb260fff] [ 0.047190] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048797] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049031] ... version: 2 [ 0.050014] ... bit width: 48 [ 0.051013] ... generic registers: 4 [ 0.052014] ... value mask: 0000ffffffffffff [ 0.053021] ... max period: 00007fffffffffff [ 0.054020] ... fixed-purpose events: 3 [ 0.055015] ... event mask: 000000070000000f [ 0.057275] rcu: Hierarchical SRCU implementation. [ 0.059600] smp: Bringing up secondary CPUs ... [ 0.060666] x86: Booting SMP configuration: [ 0.061032] .... node #0, CPUs: #1 #2 #3 [ 0.064613] smp: Brought up 1 node, 4 CPUs [ 0.066018] smpboot: Max logical packages: 1 [ 0.067025] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.226000] node 0 deferred pages initialised in 157ms [ 0.229235] devtmpfs: initialized [ 0.230235] x86/mm: Memory block size: 128MB [ 0.232833] gcov: version magic: 0x41383552 [ 0.233668] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.238086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.239372] pinctrl core: initialized pinctrl subsystem [ 0.240280] [ 0.240925] ************************************************************* [ 0.241020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.242021] ** ** [ 0.243023] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.244023] ** ** [ 0.245021] ** This means that this kernel is built to expose internal ** [ 0.246019] ** IOMMU data structures, which may compromise security on ** [ 0.247023] ** your system. ** [ 0.248019] ** ** [ 0.249020] ** If you see this message and you are not debugging the ** [ 0.250021] ** kernel, report this immediately to your vendor! ** [ 0.251019] ** ** [ 0.252018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.253021] ************************************************************* [ 0.254707] NET: Registered protocol family 16 [ 0.255484] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.256061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.257076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.258452] cpuidle: using governor menu [ 0.260481] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.263537] PCI: Using configuration type 1 for base access [ 0.265158] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.275055] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.276050] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.278044] cryptd: max_cpu_qlen set to 1000 [ 0.280230] ACPI: Added _OSI(Module Device) [ 0.281013] ACPI: Added _OSI(Processor Device) [ 0.281976] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.285021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.289123] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.294482] ACPI: Interpreter enabled [ 0.295056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.297011] ACPI: Using IOAPIC for interrupt routing [ 0.298148] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.300341] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.309872] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.311041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.313018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.315094] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.319633] acpiphp: Slot [2] registered [ 0.321145] acpiphp: Slot [5] registered [ 0.323150] acpiphp: Slot [6] registered [ 0.324090] acpiphp: Slot [7] registered [ 0.325072] acpiphp: Slot [8] registered [ 0.326073] acpiphp: Slot [9] registered [ 0.327152] acpiphp: Slot [10] registered [ 0.329134] acpiphp: Slot [3] registered [ 0.330070] acpiphp: Slot [4] registered [ 0.330900] acpiphp: Slot [11] registered [ 0.331073] acpiphp: Slot [12] registered [ 0.331922] acpiphp: Slot [13] registered [ 0.333098] acpiphp: Slot [14] registered [ 0.333961] acpiphp: Slot [15] registered [ 0.334066] acpiphp: Slot [16] registered [ 0.335025] acpiphp: Slot [17] registered [ 0.335864] acpiphp: Slot [18] registered [ 0.337059] acpiphp: Slot [19] registered [ 0.337896] acpiphp: Slot [20] registered [ 0.339093] acpiphp: Slot [21] registered [ 0.340121] acpiphp: Slot [22] registered [ 0.341065] acpiphp: Slot [23] registered [ 0.342072] acpiphp: Slot [24] registered [ 0.343018] acpiphp: Slot [25] registered [ 0.343895] acpiphp: Slot [26] registered [ 0.345097] acpiphp: Slot [27] registered [ 0.346124] acpiphp: Slot [28] registered [ 0.348124] acpiphp: Slot [29] registered [ 0.349086] acpiphp: Slot [30] registered [ 0.349937] acpiphp: Slot [31] registered [ 0.351044] PCI host bridge to bus 0000:00 [ 0.351874] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.353032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.354020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.356017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.357015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.358019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.359182] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.360717] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.362870] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.369019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.373043] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.374016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.378024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.379020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.381698] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.383604] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.385029] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.386639] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.389896] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.398014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.404013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.408098] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.414019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.421019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.441020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.449594] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.457024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.465026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.480019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.486785] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.492017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.496019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.507023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.514531] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.521021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.526020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.540023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.550199] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.554021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.560022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.573022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.584446] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.591021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.603022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.617022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.629462] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.632594] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.634419] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.637547] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.640432] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.647039] iommu: Default domain type: Passthrough [ 0.648435] SCSI subsystem initialized [ 0.650169] ACPI: bus type USB registered [ 0.651425] usbcore: registered new interface driver usbfs [ 0.654106] usbcore: registered new interface driver hub [ 0.655097] usbcore: registered new device driver usb [ 0.657226] pps_core: LinuxPPS API ver. 1 registered [ 0.659017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.662079] PTP clock support registered [ 0.665049] EDAC MC: Ver: 3.0.0 [ 0.667109] PCI: Using ACPI for IRQ routing [ 0.670020] NetLabel: Initializing [ 0.671012] NetLabel: domain hash size = 128 [ 0.673015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.675099] NetLabel: unlabeled traffic allowed by default [ 0.678144] vgaarb: loaded [ 0.679285] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.681017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.691846] clocksource: Switched to clocksource kvm-clock [ 0.803389] VFS: Disk quotas dquot_6.6.0 [ 0.804576] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.806825] *** VALIDATE ramfs *** [ 0.807819] *** VALIDATE hugetlbfs *** [ 0.809182] pnp: PnP ACPI init [ 0.811539] pnp: PnP ACPI: found 6 devices [ 0.829405] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.832359] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.834251] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.836037] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.837812] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.839645] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.842053] NET: Registered protocol family 2 [ 0.844512] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.849155] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.852383] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.857899] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.862702] TCP: Hash tables configured (established 65536 bind 65536) [ 0.865599] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.868625] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.871149] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.874111] NET: Registered protocol family 1 [ 0.877472] RPC: Registered named UNIX socket transport module. [ 0.879681] RPC: Registered udp transport module. [ 0.881531] RPC: Registered tcp transport module. [ 0.883331] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.886026] NET: Registered protocol family 44 [ 0.887814] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.889976] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.892054] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.894358] PCI: CLS 0 bytes, default 64 [ 0.896038] Unpacking initramfs... [ 2.326735] debug: unmapping init [mem 0xffff9fcc7cc54000-0xffff9fcc7ffbffff] [ 2.332701] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.334486] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.338787] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.850366] Initialise system trusted keyrings [ 2.852280] Key type blacklist registered [ 2.854294] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.863393] zbud: loaded [ 2.867963] *** VALIDATE nfs *** [ 2.869218] *** VALIDATE nfs4 *** [ 2.870945] pstore: using deflate compression [ 2.874708] Platform Keyring initialized [ 2.994931] NET: Registered protocol family 38 [ 2.996819] Key type asymmetric registered [ 2.998385] Asymmetric key parser 'x509' registered [ 3.000218] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.003296] io scheduler mq-deadline registered [ 3.007283] io scheduler kyber registered [ 3.009239] io scheduler bfq registered [ 3.011226] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.014165] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.017616] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.019914] ACPI: Power Button [PWRF] [ 3.024452] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.029534] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.041831] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.048480] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.065118] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.092612] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.122995] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.128034] Non-volatile memory driver v1.3 [ 3.129900] Linux agpgart interface v0.103 [ 3.170763] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.175272] vda: detected capacity change from 0 to 74698752 [ 3.193565] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.196788] vdb: detected capacity change from 0 to 1073741824 [ 3.213028] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.216081] vdc: detected capacity change from 0 to 2621440000 [ 3.229736] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.232136] vdd: detected capacity change from 0 to 2621440000 [ 3.246876] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.249918] vde: detected capacity change from 0 to 4294967296 [ 3.264654] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.267815] vdf: detected capacity change from 0 to 4294967296 [ 3.274505] libphy: Fixed MDIO Bus: probed [ 3.279718] usbcore: registered new interface driver usbserial_generic [ 3.282224] usbserial: USB Serial support registered for generic [ 3.284253] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.288043] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.289299] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.290841] mousedev: PS/2 mouse device common for all mice [ 3.294367] rtc_cmos 00:05: RTC can wake from S4 [ 3.294815] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.297328] rtc_cmos 00:05: registered as rtc0 [ 3.302537] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.304408] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.306118] intel_pstate: CPU model not supported [ 3.311409] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.316080] hid: raw HID events driver (C) Jiri Kosina [ 3.318249] usbcore: registered new interface driver usbhid [ 3.319648] usbhid: USB HID core driver [ 3.321128] drop_monitor: Initializing network drop monitor service [ 3.323306] Initializing XFRM netlink socket [ 3.325142] NET: Registered protocol family 10 [ 3.327603] Segment Routing with IPv6 [ 3.329110] NET: Registered protocol family 17 [ 3.331174] mpls_gso: MPLS GSO support [ 3.336799] RAS: Correctable Errors collector initialized. [ 3.339198] AVX version of gcm_enc/dec engaged. [ 3.340502] AES CTR mode by8 optimization enabled [ 3.413035] sched_clock: Marking stable (3413012881, 0)->(4353191231, -940178350) [ 3.415826] registered taskstats version 1 [ 3.417613] Loading compiled-in X.509 certificates [ 3.419294] zswap: loaded using pool lzo/zbud [ 3.444667] Key type big_key registered [ 3.456967] Key type encrypted registered [ 3.458172] ima: No TPM chip found, activating TPM-bypass! [ 3.459822] ima: Allocated hash algorithm: sha1 [ 3.461045] ima: No architecture policies found [ 3.462537] evm: Initialising EVM extended attributes: [ 3.464433] evm: security.selinux [ 3.465431] evm: security.ima [ 3.466151] evm: security.capability [ 3.467090] evm: HMAC attrs: 0x1 [ 3.469740] rtc_cmos 00:05: setting system clock to 2026-08-08 06:01:45 UTC (1786168905) [ 3.475551] debug: unmapping init [mem 0xffffffffbc203000-0xffffffffbc3fffff] [ 3.477720] debug: unmapping init [mem 0xffffffffbaf82000-0xffffffffbb258fff] [ 3.482068] Write protecting the kernel read-only data: 28672k [ 3.484779] debug: unmapping init [mem 0xffffffffb9603000-0xffffffffb97fffff] [ 3.487017] debug: unmapping init [mem 0xffffffffb9f14000-0xffffffffb9ffffff] [ 3.514887] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.520613] systemd[1]: Detected virtualization kvm. [ 3.521800] systemd[1]: Detected architecture x86-64. [ 3.523075] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.548533] systemd[1]: No hostname configured. [ 3.549771] systemd[1]: Set hostname to . [ 3.551663] random: systemd: uninitialized urandom read (16 bytes read) [ 3.554375] systemd[1]: Initializing machine ID from random generator. [ 3.600757] random: ln: uninitialized urandom read (6 bytes read) [ 3.680405] random: systemd: uninitialized urandom read (16 bytes read) [ 3.682009] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.685206] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.687976] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.258886] device-mapper: uevent: version 1.0.3 [ 4.261410] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices...[ 4.653852] random: fast init done Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.010463] virtio_net virtio0 ens2: renamed from eth0 [ 5.059490] scsi host0: ata_piix [ 5.093942] scsi host1: ata_piix [ 5.095592] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.098405] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.715602] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.976486] random: crng init done [ 9.977824] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.482944] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.693517] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.929243] SELinux: Disabled at runtime. [ 11.991112] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.999785] systemd[1]: Detected virtualization kvm. [ 12.001601] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.496731] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.500599] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.505575] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.509890] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.513837] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.524757] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.536124] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ 12.667809] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ 13.028383] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.354054] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.412144] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.552255] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.561991] EDAC sbridge: Ver: 1.1.2 [ 16.189283] Key type dns_resolver registered [ 16.995205] NFS: Registering the id_resolver key type [ 16.997511] Key type id_resolver registered [ 16.999273] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg216-server login: [ 98.256776] libcfs: loading out-of-tree module taints kernel. [ 98.316580] Key type ._llcrypt registered [ 98.318627] Key type .llcrypt registered [ 98.458902] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 118.952954] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 121.246809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 121.261306] alg: No test for adler32 (adler32-zlib) [ 122.946694] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 124.159458] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 126.175334] Key type lgssc registered [ 128.350654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 145.079849] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 195.552578] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 210.419444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 210.469536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 211.882288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 211.911228] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 212.008551] Lustre: lustre-MDT0000: new disk, initializing [ 212.087425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 212.109247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 216.875872] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.707481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 232.782692] Lustre: 6522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 232.822978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 232.827873] Lustre: Skipped 1 previous similar message [ 232.974853] Lustre: lustre-MDT0001: new disk, initializing [ 233.108331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 233.140488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 233.155947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 238.413444] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 243.436875] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 252.885976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 253.186287] Lustre: lustre-OST0000: new disk, initializing [ 253.191625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 253.194877] Lustre: 8461:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 253.289369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 258.640271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 258.659972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 258.792701] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 260.515468] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 275.978739] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 276.268526] Lustre: lustre-OST0001: new disk, initializing [ 276.278982] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 276.289218] Lustre: 9534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 276.377570] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 283.197138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 283.204827] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 283.386757] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 283.626958] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 295.644653] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 307.511725] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 314.913488] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing check_logdir /tmp/testlogs/ [ 319.524346] hrtimer: interrupt took 16334065 ns [ 322.445860] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing yml_node [ 327.148759] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 329.290891] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 332.055888] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 333.731770] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 8 02:07:14 EDT 2026 [ 351.962046] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 353.730196] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 370.657271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 370.660180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 370.674078] Lustre: Skipped 1 previous similar message [ 375.780339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 375.794804] Lustre: Skipped 6 previous similar messages [ 376.433548] Lustre: server umount lustre-MDT0000 complete [ 380.617629] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786169282 with bad export cookie 847057110589871876 [ 380.634316] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 380.915135] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 380.926415] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 380.948235] Lustre: Skipped 3 previous similar messages [ 386.021378] LustreError: 6530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 386.021411] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 386.053867] LustreError: 6530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 386.074407] Lustre: Skipped 2 previous similar messages [ 387.123876] Lustre: server umount lustre-MDT0001 complete [ 390.631347] Lustre: server umount lustre-OST0000 complete [ 394.636289] Lustre: server umount lustre-OST0001 complete [ 401.392407] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 411.384503] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 448.486205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 448.872316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 448.911911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 449.006349] Lustre: lustre-MDT0000: new disk, initializing [ 449.113660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.130517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 453.579288] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 465.166749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.332701] Lustre: 15086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 465.454997] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 465.461983] Lustre: Skipped 1 previous similar message [ 465.645291] Lustre: lustre-MDT0001: new disk, initializing [ 465.819969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 465.876462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 465.901595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 471.318717] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 481.522583] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 488.036313] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 495.287201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 495.783098] Lustre: lustre-OST0000: new disk, initializing [ 495.785802] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 495.792886] Lustre: 17056:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 495.925492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 497.853298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 497.869766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 498.070531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 503.472587] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 513.915976] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 520.016551] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 520.259203] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 526.211405] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 526.497221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 528.870605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.870898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 528.878914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 528.896606] Lustre: Skipped 1 previous similar message [ 528.937450] LustreError: Skipped 1 previous similar message [ 534.949239] Lustre: server umount lustre-OST0000 complete [ 542.692354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 542.696101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 542.706481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 542.709327] Lustre: Skipped 3 previous similar messages [ 545.733524] Lustre: server umount lustre-MDT0000 complete [ 549.345809] LustreError: 16367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.377383] LustreError: 16367:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 549.651018] LustreError: 15881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786169451 with bad export cookie 847057110589873409 [ 549.658187] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 549.661223] LustreError: 15881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 550.015215] Lustre: server umount lustre-MDT0001 complete [ 556.590940] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 02:10:57 (1786169457) [ 577.246881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 577.747056] LustreError: 19930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.834564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 582.800744] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 583.138968] LustreError: 19931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 587.319067] LustreError: 19930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 592.989323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 600.127624] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 611.324672] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 618.094880] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 620.248033] Lustre: Failing over lustre-MDT0000 [ 620.799334] Lustre: server umount lustre-MDT0000 complete [ 623.586254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 623.598398] Lustre: Skipped 1 previous similar message [ 623.603148] LustreError: 19931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 623.623251] LustreError: 19931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 639.780116] Lustre: 3642:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786169525/real 1786169525] req@ffff9fccf3cebb80 x1872933972771200/t0(0) o400->MGC192.168.202.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786169541 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 639.825838] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.841551] LustreError: 20658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 639.870460] LustreError: 20658:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 642.014383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 649.426863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 649.432090] Lustre: Skipped 1 previous similar message [ 649.506267] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 654.822634] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 654.835303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 654.857075] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 655.827267] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 665.995513] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 673.252994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 673.269904] Lustre: Skipped 1 previous similar message [ 673.275575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 673.279286] Lustre: Skipped 2 previous similar messages [ 677.222491] Lustre: server umount lustre-MDT0000 complete [ 678.382172] LustreError: 20658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 678.409676] LustreError: 20658:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 681.601768] LustreError: 19911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786169583 with bad export cookie 847057110589874823 [ 681.619457] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 681.630965] LustreError: 19911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 681.936857] Lustre: server umount lustre-MDT0001 complete [ 691.097650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.368470] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 707.699531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.793337] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 721.878860] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 728.278143] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 730.526706] Lustre: Failing over lustre-MDT0000 [ 730.835171] Lustre: server umount lustre-MDT0000 complete [ 733.607112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 733.622370] Lustre: Skipped 2 previous similar messages [ 743.908052] LustreError: 23263:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 743.935114] LustreError: 23263:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 749.024428] Lustre: 3643:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786169635/real 1786169635] req@ffff9fccf7c04380 x1872933972802304/t0(0) o400->MGC192.168.202.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786169651 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 749.060121] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 750.948903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 760.287887] LustreError: 3641:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fccf7c04a80 x1872933972805248/t0(0) o250->MGC192.168.202.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 760.656295] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 765.709500] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 765.927714] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 765.950915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 765.957057] Lustre: Skipped 1 previous similar message [ 765.976165] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 775.761112] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 781.282912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 781.289916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 781.305244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 785.831672] Lustre: server umount lustre-MDT0000 complete [ 789.347529] LustreError: 24839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786169691 with bad export cookie 847057110589875761 [ 789.355425] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 789.359667] LustreError: 24839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 789.708903] Lustre: server umount lustre-MDT0001 complete [ 797.919876] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 02:14:59 (1786169699) [ 816.485271] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 829.958059] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 870.911184] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 882.857895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 883.152821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 883.201711] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 883.341180] Lustre: lustre-MDT0000: new disk, initializing [ 883.449650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 883.454877] Lustre: Skipped 3 previous similar messages [ 883.473395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 888.939548] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 901.285230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 901.499811] Lustre: 29646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 901.784799] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 905.800763] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 914.636799] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 921.199948] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 928.299136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 928.620559] Lustre: 31616:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 930.514741] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 930.570470] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 930.666499] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 930.702231] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 935.121155] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 945.436323] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 953.199437] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 976.869125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 976.870051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 976.873324] LustreError: Skipped 1 previous similar message [ 976.880588] Lustre: Skipped 1 previous similar message [ 976.885411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 976.899766] Lustre: Skipped 3 previous similar messages [ 978.425150] Lustre: server umount lustre-OST0000 complete [ 983.523839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 997.343463] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 997.626323] Lustre: server umount lustre-MDT0000 complete [ 1001.884952] LustreError: 29638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786169903 with bad export cookie 847057110589876167 [ 1001.900380] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1002.485582] Lustre: server umount lustre-MDT0001 complete [ 1014.399858] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1017.388188] Key type lgssc unregistered [ 1017.776659] LNet: 33597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1017.779710] LNetError: 33597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1017.790673] LNet: Removed LNI 192.168.202.116@tcp [ 1018.642350] Key type .llcrypt unregistered [ 1018.644970] Key type ._llcrypt unregistered [ 1049.051618] Key type ._llcrypt registered [ 1049.053517] Key type .llcrypt registered [ 1049.187875] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1050.567444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1050.645655] alg: No test for adler32 (adler32-zlib) [ 1051.755760] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 1052.097201] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1053.799172] Key type lgssc registered [ 1055.330524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1064.182143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1064.195358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1065.719609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1070.971265] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1075.171717] LustreError: 35015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1075.190123] LustreError: 35015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1078.982879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1079.327822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1084.710910] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1092.931756] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1099.551560] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1106.386120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1106.919501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1114.364846] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1123.946500] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1124.342482] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1140.684158] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 02:20:41 (1786170041) [ 1143.056420] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1145.214950] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1152.663810] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 02:20:53 (1786170053) [ 1163.238029] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 02:21:03 (1786170063) [ 1164.428767] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1173.548694] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1173.553909] Lustre: Skipped 2 previous similar messages [ 1175.554376] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 02:21:16 (1786170076) [ 1179.589434] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1179.602028] Lustre: Skipped 3 previous similar messages [ 1200.405316] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1200.415406] Lustre: Skipped 6 previous similar messages [ 1203.789274] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 02:21:44 (1786170104) [ 1207.551261] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1210.923344] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1221.923453] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 02:22:02 (1786170122) [ 1224.201745] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1224.206871] Lustre: Skipped 3 previous similar messages [ 1238.489245] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 02:22:19 (1786170139) [ 1257.445241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.462761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1257.486263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1257.512858] Lustre: Skipped 1 previous similar message [ 1262.563253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1262.576653] Lustre: Skipped 1 previous similar message [ 1267.697078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1267.715470] Lustre: Skipped 1 previous similar message [ 1270.239152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1270.661963] Lustre: server umount lustre-OST0000 complete [ 1277.926096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1277.927629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1277.962925] Lustre: Skipped 2 previous similar messages [ 1277.983245] Lustre: Skipped 1 previous similar message [ 1283.047189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1283.053681] Lustre: Skipped 1 previous similar message [ 1289.183443] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1289.415122] Lustre: server umount lustre-MDT0000 complete [ 1293.158207] LustreError: 34995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170195 with bad export cookie 14339859382109450314 [ 1293.158901] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1293.173807] LustreError: 34995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1293.692715] Lustre: server umount lustre-MDT0001 complete [ 1307.079178] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1310.548639] Key type lgssc unregistered [ 1310.870704] LNet: 42875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1310.876925] LNetError: 42875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1310.894697] LNet: Removed LNI 192.168.202.116@tcp [ 1311.832582] Key type .llcrypt unregistered [ 1311.835078] Key type ._llcrypt unregistered [ 1332.505377] Key type ._llcrypt registered [ 1332.507474] Key type .llcrypt registered [ 1332.693837] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 1353.146049] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1354.904244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1354.931643] alg: No test for adler32 (adler32-zlib) [ 1356.033564] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 1356.315375] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1358.047985] Key type lgssc registered [ 1359.522711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1401.329672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1401.347555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1402.720696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1402.748608] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1402.872882] Lustre: lustre-MDT0000: new disk, initializing [ 1402.977963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1403.009792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1408.627866] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1421.005256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1421.170305] Lustre: 46038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1421.229474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1421.233036] Lustre: Skipped 1 previous similar message [ 1421.324369] Lustre: lustre-MDT0001: new disk, initializing [ 1421.456261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1421.501823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1421.512986] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1427.420891] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1438.755122] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1445.630712] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1453.782073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1454.073221] Lustre: lustre-OST0000: new disk, initializing [ 1454.080792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1454.091633] Lustre: 48002:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1454.280917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1461.388836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1461.405302] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1461.601706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1462.237644] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1471.978147] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1476.677364] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1476.840599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1482.171182] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1482.438844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1487.330402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1487.344512] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1487.346073] Lustre: Skipped 1 previous similar message [ 1487.349884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1487.398751] Lustre: Skipped 1 previous similar message [ 1491.260534] Lustre: server umount lustre-OST0000 complete [ 1497.577096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1497.604782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1497.613161] Lustre: Skipped 1 previous similar message [ 1501.921897] Lustre: server umount lustre-MDT0000 complete [ 1505.924719] LustreError: 46029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170407 with bad export cookie 3811215301290157020 [ 1505.949163] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1506.303671] Lustre: server umount lustre-MDT0001 complete [ 1516.418487] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 02:26:57 (1786170417) [ 1523.661773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1524.243848] LustreError: 50465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1524.302770] LustreError: 50465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1524.370154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1529.722455] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1529.830334] LustreError: 50466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1534.955410] LustreError: 50465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1540.066615] LustreError: 50466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1540.224630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1540.714745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1545.187466] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1554.736634] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1563.789944] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1574.268102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1575.008869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1584.216650] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1595.100900] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1600.659587] LustreError: 52996:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1601.587401] Lustre: Modifying parameter general.timeout=1 in log params [ 1602.698917] Lustre: Modifying parameter general.timeout=2 in log params [ 1604.795262] Lustre: Modifying parameter general.timeout=4 in log params [ 1604.800583] Lustre: Skipped 1 previous similar message [ 1609.564520] Lustre: Modifying parameter general.timeout=8 in log params [ 1609.572130] Lustre: Skipped 3 previous similar messages [ 1618.760485] Lustre: Modifying parameter general.timeout=15 in log params [ 1618.763996] Lustre: Skipped 6 previous similar messages [ 1635.438904] Lustre: Modifying parameter general.timeout=30 in log params [ 1635.442493] Lustre: Skipped 14 previous similar messages [ 1667.702024] Lustre: Modifying parameter general.timeout=56 in log params [ 1667.709673] Lustre: Skipped 25 previous similar messages [ 1732.470123] Lustre: Modifying parameter general.timeout=118 in log params [ 1732.474570] Lustre: Skipped 61 previous similar messages [ 1754.830530] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 02:30:56 (1786170656) [ 1757.739398] Lustre: 59951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1759.809114] Lustre: 60050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1761.855374] Lustre: 60148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1766.003309] Lustre: 60367:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1766.014317] Lustre: 60367:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1799.135406] Lustre: 61825:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1799.148899] Lustre: 61825:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1799.227181] LustreError: 61825:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1804.965774] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 02:31:45 (1786170705) [ 1820.642874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1820.653456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1820.654847] Lustre: Skipped 2 previous similar messages [ 1820.669582] Lustre: Skipped 1 previous similar message [ 1825.766827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1825.778841] Lustre: Skipped 2 previous similar messages [ 1827.217481] Lustre: server umount lustre-MDT0000 complete [ 1830.884166] LustreError: 50466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1830.907666] LustreError: 50466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1831.632047] LustreError: 50447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170733 with bad export cookie 3811215301290157902 [ 1831.636209] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1831.642752] LustreError: 50447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1832.288225] Lustre: server umount lustre-MDT0001 complete [ 1847.687764] Lustre: server umount lustre-OST0000 complete [ 1879.112826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1879.480878] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1879.530866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1879.550448] Lustre: Skipped 34 previous similar messages [ 1879.935402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1884.914780] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1894.834827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1895.003306] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1895.047347] Lustre: 64359:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1901.099686] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1911.526516] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1917.619048] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1925.227147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1925.577497] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1925.911476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1925.914758] Lustre: Skipped 1 previous similar message [ 1932.716576] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1939.671564] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1941.518342] Lustre: 66925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1953.254111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1953.262778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.265233] LustreError: Skipped 1 previous similar message [ 1953.266715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1953.290462] Lustre: Skipped 2 previous similar messages [ 1955.659536] Lustre: server umount lustre-OST0000 complete [ 1961.952534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1961.966503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1962.002190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1962.004068] Lustre: Skipped 1 previous similar message [ 1974.239753] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1974.786715] Lustre: server umount lustre-MDT0000 complete [ 1978.857601] LustreError: 64367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1978.885582] LustreError: 64367:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1980.731237] LustreError: 64352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170882 with bad export cookie 3811215301290161115 [ 1980.740607] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1980.761274] LustreError: 64352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1981.391139] Lustre: server umount lustre-MDT0001 complete [ 1997.760937] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2000.359263] Key type lgssc unregistered [ 2000.653702] LNet: 68139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2000.661356] LNetError: 68139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2000.680285] LNet: Removed LNI 192.168.202.116@tcp [ 2001.803422] Key type .llcrypt unregistered [ 2001.804872] Key type ._llcrypt unregistered [ 2030.472059] Key type ._llcrypt registered [ 2030.475394] Key type .llcrypt registered [ 2030.646663] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2032.012047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2032.270034] alg: No test for adler32 (adler32-zlib) [ 2033.428321] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 2033.943936] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2035.687207] Key type lgssc registered [ 2037.319757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2045.497917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2045.514833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2047.151624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2052.298268] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2057.703144] LustreError: 69553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2057.738028] LustreError: 69553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2062.527750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2062.819301] LustreError: 69554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2063.660354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2068.987494] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2077.909492] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2084.976277] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2093.534934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2094.136817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2101.478338] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2110.051827] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2128.909645] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2140.130615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2140.139959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2140.158930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2143.205163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2143.211563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2146.013935] Lustre: server umount lustre-MDT0000 complete [ 2148.328058] LustreError: 71403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2148.363907] LustreError: 71403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2150.226227] LustreError: 69534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786171052 with bad export cookie 9140121122376840312 [ 2150.230833] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2150.241926] LustreError: 69534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2150.670627] Lustre: server umount lustre-MDT0001 complete [ 2164.512930] Lustre: server umount lustre-OST0000 complete [ 2198.437431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2198.764140] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2198.792891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2199.051195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2204.482789] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2214.813971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2214.918299] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2214.944536] Lustre: 74449:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2215.321810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2220.571316] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2228.970478] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2235.544871] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2243.341175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2243.514650] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2243.895455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2251.952461] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2262.103339] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2264.072791] Lustre: 77026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2266.227358] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2273.261475] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2273.265169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2273.267610] LustreError: Skipped 1 previous similar message [ 2273.269432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2273.269438] Lustre: Skipped 2 previous similar messages [ 2273.290331] Lustre: Skipped 2 previous similar messages [ 2278.371732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2278.387091] Lustre: Skipped 1 previous similar message [ 2279.342437] Lustre: server umount lustre-OST0000 complete [ 2283.492423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2283.512968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2283.518837] Lustre: Skipped 2 previous similar messages [ 2289.282238] Lustre: server umount lustre-MDT0000 complete [ 2293.016784] LustreError: 76317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786171195 with bad export cookie 9140121122376841152 [ 2293.021593] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2293.025763] LustreError: 76317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2293.290487] Lustre: server umount lustre-MDT0001 complete [ 2309.078114] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2312.177279] Key type lgssc unregistered [ 2312.485434] LNet: 78244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2312.492428] LNetError: 78244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2312.507840] LNet: Removed LNI 192.168.202.116@tcp [ 2313.349672] Key type .llcrypt unregistered [ 2313.358034] Key type ._llcrypt unregistered [ 2342.326860] Key type ._llcrypt registered [ 2342.331989] Key type .llcrypt registered [ 2342.464710] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2343.473129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2343.488727] alg: No test for adler32 (adler32-zlib) [ 2344.886303] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 2345.152659] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2346.831223] Key type lgssc registered [ 2348.384994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2357.761347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2357.790321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2359.351570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2365.088951] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2369.514405] LustreError: 79656:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.543398] LustreError: 79656:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2374.624986] LustreError: 79657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2375.109812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2375.546582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2380.714201] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2391.602752] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2398.776240] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2407.963733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2408.936415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2415.886892] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2425.404809] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2438.148792] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2439.357544] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2440.622119] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2442.889146] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2442.896230] Lustre: Skipped 1 previous similar message [ 2447.363585] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2447.371333] Lustre: Skipped 4 previous similar messages [ 2455.609543] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2455.635794] Lustre: Skipped 6 previous similar messages [ 2471.688358] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2471.699738] Lustre: Skipped 12 previous similar messages [ 2504.510866] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2504.516270] Lustre: Skipped 29 previous similar messages [ 2556.545354] Lustre: 87789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2559.837245] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 02:44:20 (1786171460) [ 2561.568840] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2563.574586] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 02:44:24 (1786171464) [ 2567.653106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2567.656461] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2567.701589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2567.729968] Lustre: Skipped 1 previous similar message [ 2572.772034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2572.803726] Lustre: Skipped 1 previous similar message [ 2573.669230] Lustre: server umount lustre-OST0000 complete [ 2577.888769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2577.890876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2577.909701] Lustre: Skipped 2 previous similar messages [ 2583.009876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2583.012531] Lustre: Skipped 1 previous similar message [ 2583.998438] Lustre: server umount lustre-MDT0000 complete [ 2587.639303] LustreError: 79635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786171489 with bad export cookie 5312460074765485088 [ 2587.643324] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2587.651557] LustreError: 79635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2588.120433] Lustre: server umount lustre-MDT0001 complete [ 2600.268563] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2603.345159] Key type lgssc unregistered [ 2603.623633] LNet: 89047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2603.638883] LNetError: 89047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2603.655776] LNet: Removed LNI 192.168.202.116@tcp [ 2604.557320] Key type .llcrypt unregistered [ 2604.562326] Key type ._llcrypt unregistered [ 2609.703385] Key type ._llcrypt registered [ 2609.706386] Key type .llcrypt registered [ 2609.806729] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2614.856947] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules [ 2616.298407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2616.362827] alg: No test for adler32 (adler32-zlib) [ 2617.553273] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 2617.857024] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2619.567153] Key type lgssc registered [ 2621.152364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2622.670309] LustreError: 90256:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2643.989810] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2644.794895] LustreError: 90256:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2655.294792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2655.311455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2656.902913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2662.589085] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2666.483858] LustreError: 91274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2666.522331] LustreError: 91274:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2671.585480] LustreError: 91275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.445709] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 02:46:13 (1786171573) [ 2676.707541] LustreError: 91274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.257308] LustreError: 91306:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2678.270812] LustreError: 91306:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2678.281366] LustreError: 91306:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 2678.839751] Lustre: server umount lustre-MDT0000 complete [ 2695.116646] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2697.531270] Key type lgssc unregistered [ 2697.857638] LNet: 92803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2697.869120] LNetError: 92803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2697.889902] LNet: Removed LNI 192.168.202.116@tcp [ 2698.793908] Key type .llcrypt unregistered [ 2698.800451] Key type ._llcrypt unregistered [ 2718.368588] Key type ._llcrypt registered [ 2718.372632] Key type .llcrypt registered [ 2718.481459] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2719.652289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2719.684288] alg: No test for adler32 (adler32-zlib) [ 2721.180160] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 2721.558566] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2723.263283] Key type lgssc registered [ 2724.497950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2733.251889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2733.267356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2734.924062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2739.931460] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2744.305181] LustreError: 94020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.332787] LustreError: 94020:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2749.417233] LustreError: 94019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2750.011854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2751.271631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2757.399197] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2766.543531] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2772.533963] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2781.152293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2782.309486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2789.026308] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2798.093853] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2855.807296] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 02:49:16 (1786171756) [ 2857.932215] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2859.810265] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 02:49:20 (1786171760) [ 2866.658402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2866.687739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2866.689975] Lustre: Skipped 2 previous similar messages [ 2866.720369] Lustre: Skipped 2 previous similar messages [ 2871.659471] Lustre: server umount lustre-MDT0000 complete [ 2871.780657] LustreError: 94019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2871.804325] LustreError: 94019:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2875.508561] LustreError: 94001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786171777 with bad export cookie 4754952809032980177 [ 2875.510939] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2875.519362] LustreError: 94001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2875.887367] Lustre: server umount lustre-MDT0001 complete [ 2889.428454] Lustre: server umount lustre-OST0000 complete [ 2902.896142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2903.525107] LustreError: 98317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2903.778864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2908.925536] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2909.152877] LustreError: 98318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2917.394197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2918.165692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2923.017961] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2931.386366] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2938.823812] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2954.313221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2954.717660] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2954.740309] LustreError: 98305:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2954.784795] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2954.801462] LustreError: 100404:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2954.818335] LustreError: 100404:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2954.826299] LustreError: 100404:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2954.982803] Lustre: server umount lustre-OST0000 complete [ 2954.987927] LustreError: 100404:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2964.197240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2964.495987] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2964.517229] LustreError: 98307:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2964.550370] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2964.560517] LustreError: 100870:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2964.567969] LustreError: 100870:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2964.575084] LustreError: 100870:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2964.647597] Lustre: server umount lustre-OST0000 complete [ 2964.649782] LustreError: 100870:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2974.494950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2974.752635] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2974.765343] Lustre: Found index 0 for lustre-OST0000, updating log [ 2974.778768] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2974.917072] Lustre: lustre-OST0000: new disk, initializing [ 2974.923671] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2974.932704] Lustre: 101406:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2975.362296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2982.189301] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2986.126379] Lustre: Failing over lustre-OST0000 [ 2986.276384] Lustre: server umount lustre-OST0000 complete [ 2992.609696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2992.636063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2992.647985] Lustre: Skipped 1 previous similar message [ 2996.863813] Lustre: server umount lustre-MDT0000 complete [ 2997.737324] LustreError: 98317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2997.761830] LustreError: 98317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3000.469424] LustreError: 98297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786171902 with bad export cookie 4754952809032982921 [ 3000.472022] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3000.478113] LustreError: 98297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3000.815798] Lustre: server umount lustre-MDT0001 complete [ 3010.453685] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 02:51:51 (1786171911) [ 3028.051727] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3039.608482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3040.141413] LustreError: 103752:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3040.304391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3045.247838] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3054.532560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3060.964770] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3064.342955] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3070.822599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3077.845670] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3088.613336] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3088.710520] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3088.907517] Lustre: lustre-OST0001: new disk, initializing [ 3088.917318] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3088.927751] Lustre: 106069:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3089.304077] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3089.315379] Lustre: Skipped 2 previous similar messages [ 3096.909482] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3097.196892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3097.220250] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3097.384681] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3104.872863] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3115.458775] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3120.223070] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.16@tcp (stopping) [ 3122.145509] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3122.153226] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3122.162975] Lustre: Skipped 1 previous similar message [ 3123.170382] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3123.175225] Lustre: Skipped 1 previous similar message [ 3126.595612] Lustre: server umount lustre-MDT0001 complete [ 3128.302559] LustreError: 103752:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3128.325614] LustreError: 103752:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3138.177486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3138.511407] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3138.522894] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3138.636934] Lustre: 103742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3140.503188] LustreError: 107621:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3140.530237] LustreError: 107621:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3140.551590] LustreError: 103744:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3140.778838] LustreError: 107642:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.116@tcp: cfg command failed: rc = -17 [ 3140.790527] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3140.790527] [ 3145.242678] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3145.841785] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3156.512400] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 02:54:17 (1786172057) [ 3173.873718] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3191.820249] Lustre: 109685:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3214.717348] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3218.363776] Lustre: 110819:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3308.000480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3308.003660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3308.022242] Lustre: Skipped 2 previous similar messages [ 3308.027544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3308.030870] Lustre: Skipped 1 previous similar message [ 3312.463195] Lustre: server umount lustre-MDT0000 complete [ 3314.144259] LustreError: 103747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3314.161394] LustreError: 103747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 3316.480180] LustreError: 103733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786172218 with bad export cookie 4754952809032983908 [ 3316.484232] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3316.490483] LustreError: 103733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3316.963371] Lustre: server umount lustre-MDT0001 complete [ 3330.267207] Lustre: server umount lustre-OST0000 complete [ 3344.568742] Lustre: server umount lustre-OST0001 complete [ 3350.316435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3358.957864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3368.071176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3379.025407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3403.187365] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3415.173170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3415.204502] Lustre: lustre-MDT0000: reset Object Index mappings [ 3416.009576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3416.012358] Lustre: Skipped 1 previous similar message [ 3420.912438] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3430.098985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3430.131984] Lustre: lustre-MDT0001: reset Object Index mappings [ 3436.580426] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3440.121166] Lustre: 115918:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3447.652397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3455.450781] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3455.667210] LustreError: 116270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3455.679791] LustreError: 116270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3455.693229] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3460.871409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 3465.162055] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3471.368325] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3471.372113] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3472.513427] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3480.853743] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3484.290928] Lustre: 117794:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3530.777891] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 03:00:31 (1786172431) [ 3558.383887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3558.386904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3558.391905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3558.413931] Lustre: Skipped 6 previous similar messages [ 3558.445317] Lustre: Skipped 7 previous similar messages [ 3563.870535] Lustre: server umount lustre-MDT0000 complete [ 3568.157096] LustreError: 114752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786172470 with bad export cookie 4754952809033030766 [ 3568.162608] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3568.181095] LustreError: 114752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3568.614887] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3574.867482] Lustre: server umount lustre-MDT0001 complete [ 3593.184856] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3593.324651] Lustre: server umount lustre-OST0000 complete [ 3603.432825] Lustre: server umount lustre-OST0001 complete [ 3610.126226] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 3619.839625] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3656.752587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3657.052333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3657.090156] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3657.156645] Lustre: lustre-MDT0000: new disk, initializing [ 3657.240809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3657.251255] Lustre: Skipped 3 previous similar messages [ 3657.267352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3662.019917] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3670.024420] Lustre: server umount lustre-MDT0000 complete [ 3680.134276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3680.597281] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3680.622301] Lustre: 122079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3680.642657] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3686.587732] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3692.690812] Lustre: server umount lustre-MDT0000 complete [ 3702.629374] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 03:03:23 (1786172603) [ 3704.585709] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3707.116295] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 03:03:27 (1786172607) [ 3725.255274] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3737.043703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3742.393230] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3751.187313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3751.296622] Lustre: 124133:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3751.381874] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3751.385972] Lustre: Skipped 1 previous similar message [ 3751.444456] Lustre: lustre-MDT0001: new disk, initializing [ 3751.578026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3751.596732] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3756.137794] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3761.252956] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3768.358023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3768.683158] Lustre: lustre-OST0000: new disk, initializing [ 3768.687328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3768.692620] Lustre: 125711:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3770.279913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3770.291052] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3770.492406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3775.568896] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3787.561276] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3787.789719] Lustre: lustre-OST0001: new disk, initializing [ 3787.793213] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3787.800142] Lustre: 126581:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3789.019964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3789.038435] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3789.196758] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3794.774378] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3805.868451] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3810.035888] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4206.095622] Lustre: 132990:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4206.103684] Lustre: 132990:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4206.107726] Lustre: 132990:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4206.116899] Lustre: 132990:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4206.127853] Lustre: 132990:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4206.136892] Lustre: 132990:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4463.432295] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 03:16:03 (1786173363) [ 4479.457908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4479.465724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4479.489834] Lustre: Skipped 4 previous similar messages [ 4479.518555] Lustre: Skipped 11 previous similar messages [ 4484.576561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4484.597284] Lustre: Skipped 3 previous similar messages [ 4485.973607] Lustre: server umount lustre-MDT0000 complete [ 4489.716036] LustreError: 124143:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4489.743879] LustreError: 124143:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 4490.103074] LustreError: 124125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786173392 with bad export cookie 4754952809033096916 [ 4490.114364] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4490.115087] LustreError: 124125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4494.821715] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4494.825253] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4494.837497] Lustre: Skipped 1 previous similar message [ 4496.643192] Lustre: server umount lustre-MDT0001 complete [ 4515.295786] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4515.533890] Lustre: server umount lustre-OST0000 complete [ 4526.521757] Lustre: server umount lustre-OST0001 complete [ 4532.919142] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 4544.076412] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4584.905984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4585.162743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4585.200830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4585.306908] Lustre: lustre-MDT0000: new disk, initializing [ 4585.438979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4585.458441] Lustre: Skipped 5 previous similar messages [ 4585.478970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4589.856414] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4600.985041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4601.143881] Lustre: 136049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4601.237132] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4601.242951] Lustre: Skipped 1 previous similar message [ 4601.336930] Lustre: lustre-MDT0001: new disk, initializing [ 4601.491305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4601.513033] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4606.854751] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4617.568586] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4623.883329] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4631.260433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4631.467715] Lustre: lustre-OST0000: new disk, initializing [ 4631.472299] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4631.484784] Lustre: 138016:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4631.577281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4631.586163] Lustre: Skipped 1 previous similar message [ 4632.861736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4632.875896] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4632.962414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4637.697256] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4647.893929] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4659.070973] Lustre: lustre-MDD0000: changelog on [ 4661.634720] Lustre: lustre-MDD0001: changelog on [ 5142.766323] Lustre: lustre-MDD0001: changelog off [ 5143.359865] Lustre: 136056:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5143.649776] Lustre: lustre-MDD0000: changelog off [ 5150.180644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5150.181615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5150.184876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5150.187848] Lustre: Skipped 1 previous similar message [ 5150.189598] Lustre: Skipped 3 previous similar messages [ 5155.301487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5155.305313] Lustre: Skipped 2 previous similar messages [ 5159.391128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5159.458157] Lustre: server umount lustre-OST0000 complete [ 5164.512495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5164.518775] LustreError: Skipped 1 previous similar message [ 5164.522504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5164.532838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5165.537834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5166.910773] Lustre: server umount lustre-MDT0000 complete [ 5168.145684] LustreError: 136040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174070 with bad export cookie 4754952809033870773 [ 5168.149462] LustreError: 136040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5168.150207] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5168.288210] Lustre: server umount lustre-MDT0001 complete [ 5171.899172] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5172.881883] Key type lgssc unregistered [ 5173.009350] LNet: 146496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5173.011403] LNetError: 146496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5173.022271] LNet: Removed LNI 192.168.202.116@tcp [ 5173.340127] Key type .llcrypt unregistered [ 5173.341396] Key type ._llcrypt unregistered [ 5179.843848] Key type ._llcrypt registered [ 5179.844965] Key type .llcrypt registered [ 5179.882725] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 5186.271553] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5186.605791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5186.639900] alg: No test for adler32 (adler32-zlib) [ 5187.495623] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 5187.582262] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5189.159168] Key type lgssc registered [ 5189.528383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5200.442332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5200.446735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5201.528796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5201.537438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5201.574047] Lustre: lustre-MDT0000: new disk, initializing [ 5201.593526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5201.599164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5202.885332] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5206.960638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5206.999556] Lustre: 149671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5207.021437] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5207.023265] Lustre: Skipped 1 previous similar message [ 5207.058025] Lustre: lustre-MDT0001: new disk, initializing [ 5207.087377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5207.099119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5207.103929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5208.354806] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5212.289652] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5214.230516] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5216.495857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5216.590708] Lustre: lustre-OST0000: new disk, initializing [ 5216.592224] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5216.594485] Lustre: 151636:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5216.620471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5217.548938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5217.555806] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5217.608191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5218.605975] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5222.222718] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5224.429491] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_module obdecho/obdecho [ 5225.013643] LustreError: 152640:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5225.015971] LustreError: 152640:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5225.570204] LustreError: 152736:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5232.608337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5232.611856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5232.618154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5233.122621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5233.124350] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5233.130668] Lustre: Skipped 1 previous similar message [ 5234.920003] Lustre: server umount lustre-MDT0000 complete [ 5236.177487] LustreError: 149661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174138 with bad export cookie 2897001061679279698 [ 5236.180426] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5236.182528] LustreError: 149661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5236.299986] Lustre: server umount lustre-MDT0001 complete [ 5247.854121] Lustre: server umount lustre-OST0000 complete [ 5255.267613] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 5258.737595] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5272.660196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5272.769371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5272.782132] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5272.827914] Lustre: lustre-MDT0000: new disk, initializing [ 5272.854224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5272.860895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5274.184945] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5278.614619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5278.657497] Lustre: 156441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5278.680232] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5278.682247] Lustre: Skipped 1 previous similar message [ 5278.721843] Lustre: lustre-MDT0001: new disk, initializing [ 5278.756023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5278.774377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5278.778063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5280.077871] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5284.243509] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5286.106017] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5288.467282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5288.586443] Lustre: lustre-OST0000: new disk, initializing [ 5288.587953] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5288.590483] Lustre: 158409:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5288.627297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5290.668335] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5294.081936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5294.085060] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5294.126671] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5294.510282] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5296.005800] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5296.093713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5297.475368] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5297.554024] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5299.172089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5299.172165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5299.177139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5299.178587] LustreError: Skipped 1 previous similar message [ 5299.182630] Lustre: Skipped 3 previous similar messages [ 5304.277539] Lustre: server umount lustre-OST0000 complete [ 5309.411788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5309.411942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5309.412805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5309.428098] Lustre: Skipped 2 previous similar messages [ 5311.662750] Lustre: server umount lustre-MDT0000 complete [ 5312.871129] LustreError: 156435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174214 with bad export cookie 2897001061679280643 [ 5312.874557] LustreError: 156435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5312.874655] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5312.996929] Lustre: server umount lustre-MDT0001 complete [ 5314.211766] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 03:30:16 (1786174216) [ 5314.771829] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5315.321334] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 03:30:17 (1786174217) [ 5317.821611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5318.026465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5319.271467] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5322.029591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5323.556741] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5326.018241] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5327.951486] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5330.784539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5332.822651] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5335.206184] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5347.808322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5347.811040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5347.816816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5347.819038] Lustre: Skipped 1 previous similar message [ 5353.777226] Lustre: server umount lustre-MDT0000 complete [ 5355.103229] LustreError: 160798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174257 with bad export cookie 2897001061679281525 [ 5355.108171] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5355.109779] LustreError: 160798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5356.078938] LustreError: 160813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5356.085213] LustreError: 160813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5356.086641] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.16@tcp (stopping) [ 5356.092046] Lustre: Skipped 7 previous similar messages [ 5358.559756] LustreError: 160812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5358.560048] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5358.568493] Lustre: Skipped 2 previous similar messages [ 5360.607821] LustreError: 160813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5361.283724] Lustre: server umount lustre-MDT0001 complete [ 5363.441024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5367.914634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5367.991274] LustreError: 164674:0:(llog.c:1655:llog_backup()) MGC192.168.202.116@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5367.994148] Lustre: 164674:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.116@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5385.183855] LustreError: 164679:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5385.188336] LustreError: 164679:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5390.303966] LustreError: 164680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5390.309564] LustreError: 164680:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5400.544068] LustreError: 164700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5400.549416] LustreError: 164700:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 5418.975970] LustreError: 164700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5418.984064] LustreError: 164700:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 5428.831397] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5428.835456] Lustre: 164674:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.116@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5452.334405] LustreError: 164681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5452.339065] LustreError: 164681:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [ 5469.679312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5469.683669] Lustre: Skipped 2 previous similar messages [ 5471.178649] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5474.774329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5474.789852] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5475.808368] LustreError: 147895:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fb795000 x1872939281645952/t0(0) o250->MGC192.168.202.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5477.542651] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5480.194532] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5480.735561] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5481.456255] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5483.095610] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5483.733768] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5491.682182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5491.685124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5491.690467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5491.697912] Lustre: Skipped 1 previous similar message [ 5491.703208] Lustre: Skipped 3 previous similar messages [ 5496.121802] Lustre: server umount lustre-MDT0000 complete [ 5497.361467] LustreError: 160798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174399 with bad export cookie 2897001061679287965 [ 5497.364788] LustreError: 160798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5497.365338] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5497.476456] Lustre: server umount lustre-MDT0001 complete [ 5508.053284] Lustre: server umount lustre-OST0000 complete [ 5516.456739] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 5519.805255] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5534.600589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5534.719576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5534.732250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5534.786610] Lustre: lustre-MDT0000: new disk, initializing [ 5534.813596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5534.818182] Lustre: Skipped 1 previous similar message [ 5534.827075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5536.234699] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5541.323675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5541.371220] Lustre: 169748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5541.474581] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5542.785319] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5547.249492] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5549.673610] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5552.714277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5552.875809] Lustre: 171720:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5555.175394] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5559.168859] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5560.311790] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5560.347125] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5560.897273] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5560.987250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5562.774493] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5562.887791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5565.411274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5565.412084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5565.413966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5565.414790] LustreError: Skipped 1 previous similar message [ 5565.421193] Lustre: Skipped 2 previous similar messages [ 5569.884966] Lustre: server umount lustre-OST0000 complete [ 5577.379356] Lustre: server umount lustre-MDT0000 complete [ 5578.849390] LustreError: 169740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174480 with bad export cookie 2897001061679304765 [ 5578.855921] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5578.857068] LustreError: 169740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5579.003362] Lustre: server umount lustre-MDT0001 complete [ 5582.502640] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 03:34:44 (1786174484) [ 5585.164722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5585.363987] LustreError: 174132:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5585.370348] LustreError: 174132:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 22 previous similar messages [ 5587.041424] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5590.005745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5591.998574] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5594.836932] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5597.066140] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5600.014900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5602.206342] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5605.271689] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5614.597197] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 03:35:16 (1786174516) [ 5616.608640] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5616.611949] LustreError: Skipped 1 previous similar message [ 5621.741753] Lustre: server umount lustre-OST0000 complete [ 5629.804231] Lustre: server umount lustre-MDT0000 complete [ 5631.350119] LustreError: 174113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174533 with bad export cookie 2897001061679305633 [ 5631.355870] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5631.514923] Lustre: server umount lustre-MDT0001 complete [ 5636.091281] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5637.341249] Key type lgssc unregistered [ 5637.469412] LNet: 177857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5637.473376] LNetError: 177857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5637.482316] LNet: Removed LNI 192.168.202.116@tcp [ 5637.912171] Key type .llcrypt unregistered [ 5637.915107] Key type ._llcrypt unregistered [ 5646.191022] Key type ._llcrypt registered [ 5646.192065] Key type .llcrypt registered [ 5646.232854] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5646.651866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5646.661110] alg: No test for adler32 (adler32-zlib) [ 5647.513598] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 5647.598766] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5649.175171] Key type lgssc registered [ 5649.590557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5652.234364] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5652.241830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5667.679547] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5672.799311] LustreError: 179077:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 5677.919721] LustreError: 179077:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5677.928065] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5677.932254] LustreError: MGC192.168.202.116@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5677.940346] LustreError: 179077:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5677.945641] LustreError: 179077:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 5677.949792] LustreError: 179077:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5677.975898] Lustre: server umount lustre-OST0000 complete [ 5677.977913] LustreError: 179077:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5681.717560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5682.875213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5684.176528] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5688.084795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5688.253712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5689.733109] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5693.082864] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5695.323655] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5697.996623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5698.081111] LustreError: 181389:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5698.084721] Lustre: 181389:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.116@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5698.131497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5700.573990] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5703.344200] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5704.673736] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5704.674539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5704.677519] Lustre: Skipped 1 previous similar message [ 5704.682866] Lustre: Skipped 2 previous similar messages [ 5709.795613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5709.798300] Lustre: Skipped 2 previous similar messages [ 5710.434211] Lustre: server umount lustre-MDT0000 complete [ 5712.247071] LustreError: 179556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174614 with bad export cookie 987701134962745628 [ 5712.251947] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5712.255166] LustreError: 179556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5712.393012] Lustre: server umount lustre-MDT0001 complete [ 5723.901120] Lustre: server umount lustre-OST0000 complete [ 5728.205023] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 5732.029844] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5748.933805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5749.037377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5749.049052] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5749.088180] Lustre: lustre-MDT0000: new disk, initializing [ 5749.110334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5749.117469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5750.449718] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5755.350725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5755.393494] Lustre: 184905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5755.419711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5755.422177] Lustre: Skipped 1 previous similar message [ 5755.459462] Lustre: lustre-MDT0001: new disk, initializing [ 5755.498266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5755.513078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5755.516449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5757.027958] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5761.057759] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5763.328430] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5766.346738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5766.491762] Lustre: lustre-OST0000: new disk, initializing [ 5766.495305] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5766.500369] Lustre: 186875:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5766.541442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5769.015846] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5769.426888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5769.432944] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5769.483866] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5773.543201] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5775.125712] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5775.227737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5776.969885] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5777.045212] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5779.941260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5779.941394] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5779.945568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5779.946908] Lustre: Skipped 2 previous similar messages [ 5779.951564] LustreError: Skipped 1 previous similar message [ 5785.063258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5785.067604] Lustre: Skipped 2 previous similar messages [ 5790.180759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5790.184821] Lustre: Skipped 1 previous similar message [ 5792.223083] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5792.285273] Lustre: server umount lustre-OST0000 complete [ 5795.298582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5795.306607] Lustre: Skipped 1 previous similar message [ 5800.047747] Lustre: server umount lustre-MDT0000 complete [ 5800.416178] LustreError: 184918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5800.421074] LustreError: 184918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5801.922814] LustreError: 185700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174703 with bad export cookie 987701134962746328 [ 5801.930436] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5801.930770] LustreError: 185700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5802.103664] Lustre: server umount lustre-MDT0001 complete [ 5806.110166] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 03:38:28 (1786174708) [ 5813.771697] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5815.100638] Key type lgssc unregistered [ 5815.234487] LNet: 189821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5815.236643] LNetError: 189821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5815.246423] LNet: Removed LNI 192.168.202.116@tcp [ 5815.573912] Key type .llcrypt unregistered [ 5815.574959] Key type ._llcrypt unregistered [ 5823.997941] Key type ._llcrypt registered [ 5823.999193] Key type .llcrypt registered [ 5824.060660] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5824.614240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5824.622849] alg: No test for adler32 (adler32-zlib) [ 5825.505086] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 5825.596941] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5827.183294] Key type lgssc registered [ 5827.614250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5830.905007] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5830.914742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5832.106984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5833.784392] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5837.689297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5837.867882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5839.791244] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5843.192749] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5846.028624] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5849.938951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5850.040572] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5850.042844] LustreError: 192878:0:(llog.c:1696:llog_backup()) MGC192.168.202.116@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5850.047430] Lustre: 192878:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.116@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5850.105137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5852.863321] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5856.630719] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5858.272816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5858.278392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5858.287940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5860.320932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5860.322700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5862.369202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5862.372333] Lustre: Skipped 2 previous similar messages [ 5867.491028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5867.494093] Lustre: Skipped 1 previous similar message [ 5872.095308] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5872.189855] Lustre: server umount lustre-MDT0000 complete [ 5872.613529] LustreError: 191012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5872.622935] LustreError: 191012:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5873.823073] LustreError: 190998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174775 with bad export cookie 3394726048358009333 [ 5873.826224] LustreError: 190998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5873.826843] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5874.655334] LustreError: 191746:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009a499100 x1872939950561280/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5874.739627] Lustre: server umount lustre-MDT0001 complete [ 5886.555371] Lustre: server umount lustre-OST0000 complete [ 5890.710397] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 5894.926750] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5910.003091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5910.132220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5910.150314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5910.198851] Lustre: lustre-MDT0000: new disk, initializing [ 5910.236825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5910.249969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5911.760449] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5917.312673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5917.372183] Lustre: 196397:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5917.398720] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5917.401134] Lustre: Skipped 1 previous similar message [ 5917.402550] LustreError: 197179:0:(llog.c:1696:llog_backup()) MGC192.168.202.116@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5917.407770] LustreError: 197179:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5917.411134] Lustre: 197179:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.116@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5917.423740] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5917.426527] Lustre: Skipped 1 previous similar message [ 5917.469458] Lustre: lustre-MDT0001: new disk, initializing [ 5917.515348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5917.533158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5917.536987] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5919.316431] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5924.287409] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5926.951341] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5930.433176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5930.617217] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5930.619830] Lustre: Skipped 1 previous similar message [ 5930.622346] LustreError: 198344:0:(llog.c:1696:llog_backup()) MGC192.168.202.116@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5930.627698] LustreError: 198344:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5930.631735] Lustre: 198344:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.116@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5930.638260] Lustre: 198344:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5930.654140] Lustre: lustre-OST0000: new disk, initializing [ 5930.656975] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5930.661258] Lustre: 198367:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5930.711938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5933.190251] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5937.447924] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5938.685980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5938.689558] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5938.737098] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5939.241395] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5939.318588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5941.107167] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5941.180159] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5943.778766] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5943.779072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5943.781566] LustreError: Skipped 1 previous similar message [ 5943.782211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5943.785202] Lustre: Skipped 2 previous similar messages [ 5943.794322] Lustre: Skipped 1 previous similar message [ 5948.011763] Lustre: server umount lustre-OST0000 complete [ 5953.504845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5953.512300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5953.521344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5964.255278] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5964.353919] Lustre: server umount lustre-MDT0000 complete [ 5966.271873] LustreError: 196390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174868 with bad export cookie 3394726048358010033 [ 5966.277844] LustreError: 196390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5966.278880] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5966.481738] Lustre: server umount lustre-MDT0001 complete [ 5970.261508] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 03:41:12 (1786174872) [ 5978.505321] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5982.800856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5982.969674] LustreError: 201488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5982.985715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5984.770250] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5988.322831] LustreError: 201489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5989.104622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5991.227692] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5992.710943] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5996.045573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5998.695060] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6002.983970] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6003.095397] Lustre: lustre-OST0001: new disk, initializing [ 6003.098704] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6003.103292] Lustre: 203796:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6005.827215] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6008.837450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6008.840963] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6008.876525] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6016.370998] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6020.826418] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6021.783363] Lustre: DEBUG MARKER: ADD OST3 [ 6024.099117] loop: module loaded [ 6024.134066] loop0: detected capacity change from 0 to 204800000 [ 6024.286190] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6024.289795] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6024.294262] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6027.134753] loop0: detected capacity change from 0 to 204800000 [ 6029.640153] Lustre: DEBUG MARKER: STOP OST3 [ 6029.679498] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6029.775906] Lustre: lustre-OST0003: new disk, initializing [ 6029.777561] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 6029.779948] Lustre: 205581:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6029.819896] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 6029.822406] Lustre: Skipped 3 previous similar messages [ 6030.468447] Lustre: Failing over lustre-OST0003 [ 6030.508805] Lustre: server umount lustre-OST0003 complete [ 6030.781811] LustreError: 202974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6032.211756] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6036.025811] LustreError: 204319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6036.037462] LustreError: 204319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6038.046729] Lustre: DEBUG MARKER: START OST3 again [ 6041.032743] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6043.990213] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6046.208642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 6046.216538] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 6046.233855] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 6058.211258] Lustre: Failing over lustre-OST0003 [ 6058.260559] Lustre: server umount lustre-OST0003 complete [ 6061.544085] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6061.547220] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6061.549095] LustreError: Skipped 1 previous similar message [ 6061.550357] LustreError: 202974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6061.557277] Lustre: Skipped 2 previous similar messages [ 6061.560258] LustreError: 202974:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6071.265155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6071.270716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6071.280378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6071.284463] Lustre: Skipped 6 previous similar messages [ 6077.314173] Lustre: server umount lustre-MDT0000 complete [ 6079.037646] LustreError: 201471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174981 with bad export cookie 3394726048358010915 [ 6079.045278] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6079.051833] LustreError: 201471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6082.017858] LustreError: 202993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6082.031910] LustreError: 202993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6085.337845] Lustre: server umount lustre-MDT0001 complete [ 6087.231389] Lustre: server umount lustre-OST0000 complete [ 6089.094346] Lustre: server umount lustre-OST0001 complete [ 6092.412361] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 6096.632110] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6114.629362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6114.742324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6114.756858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6114.809443] Lustre: lustre-MDT0000: new disk, initializing [ 6114.836779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6114.839775] Lustre: Skipped 1 previous similar message [ 6114.847068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6116.512583] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6121.482671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6121.525792] Lustre: 211093:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6121.661897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6123.220360] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6127.802690] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6130.504990] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6133.890897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6134.066056] Lustre: lustre-OST0000: new disk, initializing [ 6134.067927] Lustre: Skipped 1 previous similar message [ 6134.069795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6134.072118] Lustre: Skipped 2 previous similar messages [ 6134.074518] Lustre: 213062:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6135.368334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6135.371218] Lustre: Skipped 1 previous similar message [ 6135.373884] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6135.423216] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6136.325306] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6141.207614] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6142.931589] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6143.018550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6145.067084] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6145.202365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6151.143503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6151.144646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6151.148780] LustreError: Skipped 1 previous similar message [ 6151.157479] Lustre: Skipped 5 previous similar messages [ 6151.161671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6151.167430] Lustre: Skipped 12 previous similar messages [ 6160.351169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6160.425338] Lustre: server umount lustre-OST0000 complete [ 6168.378191] Lustre: server umount lustre-MDT0000 complete [ 6170.136769] LustreError: 211899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786175072 with bad export cookie 3394726048358013162 [ 6170.144331] LustreError: 211899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6170.145355] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6170.310650] Lustre: server umount lustre-MDT0001 complete [ 6171.841767] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 03:44:33 (1786175073) [ 6179.562470] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 6184.000207] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6201.371613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6201.508681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6201.523524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6201.577361] Lustre: lustre-MDT0000: new disk, initializing [ 6201.609844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6203.525636] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6208.537317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6208.577621] Lustre: 217663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6208.707970] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6210.167598] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6214.760847] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6217.211328] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6220.586958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6220.752302] Lustre: 219632:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6222.449340] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6223.418842] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6228.063478] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6229.997479] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6230.073904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6231.889751] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6231.971559] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6233.059747] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6233.059949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6233.064091] LustreError: Skipped 1 previous similar message [ 6233.071251] Lustre: Skipped 2 previous similar messages [ 6233.072326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6233.080019] Lustre: Skipped 6 previous similar messages [ 6246.879054] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6246.933026] Lustre: server umount lustre-OST0000 complete [ 6254.296900] Lustre: server umount lustre-MDT0000 complete [ 6255.486590] LustreError: 219631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786175157 with bad export cookie 3394726048358014044 [ 6255.490250] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6255.492816] LustreError: 219631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6255.611729] Lustre: server umount lustre-MDT0001 complete [ 6258.360856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6258.518015] LustreError: 221846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6258.523483] LustreError: 221846:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6258.541551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6258.543654] Lustre: Skipped 5 previous similar messages [ 6259.782690] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6263.519876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6265.144802] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6268.292302] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6271.048879] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6274.264214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6277.114143] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6279.911483] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6293.989922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6293.995769] LustreError: Skipped 1 previous similar message [ 6298.965988] Lustre: server umount lustre-OST0000 complete [ 6304.225465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6304.229664] Lustre: Skipped 6 previous similar messages [ 6306.432135] Lustre: server umount lustre-MDT0000 complete [ 6308.108154] LustreError: 222582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786175210 with bad export cookie 3394726048358014926 [ 6308.114951] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6308.115878] LustreError: 222582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6308.258465] Lustre: server umount lustre-MDT0001 complete [ 6413.284600] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6414.276978] Key type lgssc unregistered [ 6414.405305] LNet: 225532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6414.407374] LNetError: 225532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6414.414376] LNet: Removed LNI 192.168.202.116@tcp [ 6414.699100] Key type .llcrypt unregistered [ 6414.700118] Key type ._llcrypt unregistered [ 6423.851055] Key type ._llcrypt registered [ 6423.852101] Key type .llcrypt registered [ 6423.900813] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 6431.150172] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6431.514557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6431.523765] alg: No test for adler32 (adler32-zlib) [ 6432.375159] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 6432.470181] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6434.055119] Key type lgssc registered [ 6434.413235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6447.110795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6447.117049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6448.237622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6448.254657] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6448.307355] Lustre: lustre-MDT0000: new disk, initializing [ 6448.342893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6448.351145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6449.801445] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6454.495753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6454.547567] Lustre: 228857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6454.582796] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6454.585752] Lustre: Skipped 1 previous similar message [ 6454.631634] Lustre: lustre-MDT0001: new disk, initializing [ 6454.685781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6454.705846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6454.712089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6456.316990] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6460.489926] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6462.497170] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6464.886368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6465.008127] Lustre: lustre-OST0000: new disk, initializing [ 6465.009913] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6465.012465] Lustre: 230826:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6465.043504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6465.138441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6465.141489] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6465.182607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6467.117473] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6471.234135] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6473.156235] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6473.245501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6474.861412] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6474.951415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6475.744767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6475.746366] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6475.747098] LustreError: Skipped 1 previous similar message [ 6475.750829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6475.753184] Lustre: Skipped 1 previous similar message [ 6480.865827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6480.867653] Lustre: Skipped 1 previous similar message [ 6485.985943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6485.988206] Lustre: Skipped 1 previous similar message [ 6490.079289] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6490.141346] Lustre: server umount lustre-OST0000 complete [ 6495.712515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6495.714828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6495.718107] Lustre: Skipped 1 previous similar message [ 6495.720159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6497.884392] Lustre: server umount lustre-MDT0000 complete [ 6499.169372] LustreError: 228852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786175401 with bad export cookie 17158836369222402191 [ 6499.172176] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6499.172666] LustreError: 228852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6499.294154] Lustre: server umount lustre-MDT0001 complete [ 6505.054028] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6508.564982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6508.705494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6509.969831] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6512.722617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6512.838154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6514.153697] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6515.135763] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6517.771984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6520.400413] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6523.429343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6523.495321] Lustre: lustre-OST0001: new disk, initializing [ 6523.496837] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6523.499232] Lustre: 236079:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6523.535502] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6523.537410] Lustre: Skipped 1 previous similar message [ 6525.534614] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6530.047401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6530.050836] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6530.082523] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6531.364084] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6537.249609] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6541.401434] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 03:50:43 (1786175443) [ 6543.840188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6543.842909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6543.847153] Lustre: Skipped 1 previous similar message [ 6543.849223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6543.851313] Lustre: Skipped 2 previous similar messages [ 6548.288023] Lustre: server umount lustre-MDT0000 complete [ 6549.801222] LustreError: 236078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786175451 with bad export cookie 17158836369222403059 [ 6549.805015] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6549.805649] LustreError: 236078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6549.933272] Lustre: server umount lustre-MDT0001 complete [ 6561.555696] Lustre: server umount lustre-OST0000 complete [ 6573.444541] Lustre: server umount lustre-OST0001 complete [ 6576.368824] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 6580.590579] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6594.010070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6594.121413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6594.132195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6594.173466] Lustre: lustre-MDT0000: new disk, initializing [ 6594.196758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6594.204192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6595.688896] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6599.829812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6599.873271] Lustre: 240015:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6599.989334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6601.327224] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6605.401909] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6607.391818] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6609.874480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6609.993810] Lustre: lustre-OST0000: new disk, initializing [ 6609.995382] Lustre: Skipped 1 previous similar message [ 6609.996808] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6609.998514] Lustre: Skipped 2 previous similar messages [ 6610.000652] Lustre: 241983:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6611.665212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6611.669573] Lustre: Skipped 1 previous similar message [ 6611.673123] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6611.724212] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6612.229927] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6616.385570] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6617.822810] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6617.903485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6619.532127] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6619.620173] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6622.183188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6622.183716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6622.189062] LustreError: Skipped 1 previous similar message [ 6622.197516] Lustre: Skipped 3 previous similar messages [ 6622.198703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6622.209690] Lustre: Skipped 5 previous similar messages [ 6626.464203] Lustre: server umount lustre-OST0000 complete [ 6630.880662] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6630.885344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6630.892665] Lustre: Skipped 1 previous similar message [ 6634.103853] Lustre: server umount lustre-MDT0000 complete [ 6635.660374] LustreError: 240007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786175537 with bad export cookie 17158836369222405180 [ 6635.664162] LustreError: 240007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6635.664370] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6635.814798] Lustre: server umount lustre-MDT0001 complete [ 6639.525742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6639.697212] LustreError: 244193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6639.703608] LustreError: 244193:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6639.732388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6639.735829] Lustre: Skipped 2 previous similar messages [ 6641.704825] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6645.186843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6645.216830] LustreError: 244194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6647.007088] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6650.005158] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6652.103538] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6654.649275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6657.317721] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6659.957881] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6804.449239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6804.455080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6804.463908] Lustre: Skipped 1 previous similar message [ 6804.467717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6804.471480] Lustre: Skipped 3 previous similar messages [ 6810.190576] Lustre: server umount lustre-OST0000 complete [ 6817.842457] Lustre: server umount lustre-MDT0000 complete [ 6819.488685] LustreError: 244176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786175721 with bad export cookie 17158836369222406062 [ 6819.493449] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6819.626982] Lustre: server umount lustre-MDT0001 complete [ 6925.648125] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6926.897594] Key type lgssc unregistered [ 6927.047710] LNet: 247805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6927.053307] LNetError: 247805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6927.063609] LNet: Removed LNI 192.168.202.116@tcp [ 6927.480152] Key type .llcrypt unregistered [ 6927.482043] Key type ._llcrypt unregistered [ 6937.629593] Key type ._llcrypt registered [ 6937.630583] Key type .llcrypt registered [ 6937.665348] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 6944.561347] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6944.987973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6944.995809] alg: No test for adler32 (adler32-zlib) [ 6945.844763] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 6945.929836] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6947.511110] Key type lgssc registered [ 6947.908096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6964.451205] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6968.587372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6968.592686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6969.709568] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6969.724850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6969.798553] Lustre: lustre-MDT0000: new disk, initializing [ 6969.824537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6969.832895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6971.386606] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6976.163781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6976.212477] Lustre: 251846:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6976.245572] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6976.249314] Lustre: Skipped 1 previous similar message [ 6976.289304] Lustre: lustre-MDT0001: new disk, initializing [ 6976.332909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6976.346335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6976.351843] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6977.886936] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6980.241804] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6982.703403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6982.836482] Lustre: lustre-OST0000: new disk, initializing [ 6982.838421] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6982.841151] Lustre: 253474:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6982.877977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6985.049235] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6985.856087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6985.859712] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6985.904480] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6989.675743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6989.785414] Lustre: lustre-OST0001: new disk, initializing [ 6989.787670] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6989.790761] Lustre: 254348:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6989.854891] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6992.052474] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6993.752465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6993.757108] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6993.797714] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6998.071912] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7000.889185] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7004.133326] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7004.136469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7004.137024] Lustre: Skipped 3 previous similar messages [ 7004.139378] Lustre: Skipped 2 previous similar messages [ 7006.130490] Lustre: server umount lustre-MDT0000 complete [ 7007.446681] LustreError: 252640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786175909 with bad export cookie 4634252967619226067 [ 7007.449676] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7007.450295] LustreError: 252640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7009.250640] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7009.259040] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7009.262972] Lustre: Skipped 1 previous similar message [ 7009.607837] Lustre: server umount lustre-MDT0001 complete [ 7010.988607] Lustre: server umount lustre-OST0000 complete [ 7014.462191] Lustre: server umount lustre-OST0001 complete [ 7019.600858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7022.921628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7025.999449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7029.827182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7036.954795] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7040.809274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7040.819151] Lustre: lustre-MDT0000: reset Object Index mappings [ 7041.002395] LustreError: 259345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7041.008337] LustreError: 259345:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7041.022715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7042.737974] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7046.113771] LustreError: 259346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7046.466558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7046.478923] Lustre: lustre-MDT0001: reset Object Index mappings [ 7048.161332] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7049.490655] Lustre: 260487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7052.257841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7052.437685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7052.439916] Lustre: Skipped 1 previous similar message [ 7053.483832] LustreError: 261018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7053.492396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7054.650168] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7056.546177] LustreError: 260844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7056.551538] LustreError: 260844:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7057.592820] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7060.108522] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7063.021881] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 7063.812985] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7065.111656] Lustre: 262357:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7067.104989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7067.109992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7067.117681] Lustre: Skipped 1 previous similar message [ 7067.121328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7072.517602] Lustre: server umount lustre-MDT0000 complete [ 7073.253126] LustreError: 259346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7073.263230] LustreError: 259346:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7074.280510] LustreError: 259327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786175976 with bad export cookie 4634252967619232850 [ 7074.287115] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7074.292582] LustreError: 259327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7074.450712] Lustre: server umount lustre-MDT0001 complete [ 7086.611732] Lustre: server umount lustre-OST0000 complete [ 7098.709093] Lustre: server umount lustre-OST0001 complete [ 7106.317398] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 04:00:08 (1786176008) [ 7112.762926] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 7116.653515] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7131.332451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7131.444831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7131.459156] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7131.505131] Lustre: lustre-MDT0000: new disk, initializing [ 7131.534187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7131.537760] Lustre: Skipped 1 previous similar message [ 7131.545652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7133.340954] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7138.232381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7138.278977] Lustre: 266607:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7138.408320] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7139.999317] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7144.380343] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7146.322777] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7149.557481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7149.710810] Lustre: lustre-OST0000: new disk, initializing [ 7149.712860] Lustre: Skipped 1 previous similar message [ 7149.715184] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7149.717690] Lustre: Skipped 2 previous similar messages [ 7149.720521] Lustre: 268576:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7151.766388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7151.770824] Lustre: Skipped 1 previous similar message [ 7151.774158] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7151.825527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7152.703456] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7157.282666] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7158.892283] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7158.972373] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7160.668371] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7160.761732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7162.344426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7162.345236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7162.349573] LustreError: Skipped 1 previous similar message [ 7162.357130] Lustre: Skipped 3 previous similar messages [ 7162.358234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7162.366480] Lustre: Skipped 5 previous similar messages [ 7167.459173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7167.461200] Lustre: Skipped 1 previous similar message [ 7167.583137] Lustre: server umount lustre-OST0000 complete [ 7172.577934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7172.583316] Lustre: Skipped 2 previous similar messages [ 7177.697235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7177.700204] Lustre: Skipped 3 previous similar messages [ 7183.327098] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7183.422521] Lustre: server umount lustre-MDT0000 complete [ 7185.121831] LustreError: 266599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176087 with bad export cookie 4634252967619234936 [ 7185.128525] LustreError: 266599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7185.129543] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7185.263979] Lustre: server umount lustre-MDT0001 complete [ 7192.329672] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7196.418293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7196.571883] LustreError: 271516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7196.586636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7196.589018] Lustre: Skipped 2 previous similar messages [ 7198.122354] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7201.088149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7202.591860] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7203.596254] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7206.141426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7208.431269] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7211.514725] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7211.607939] Lustre: lustre-OST0001: new disk, initializing [ 7211.610109] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7211.613714] Lustre: 273825:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7212.672996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7212.676335] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7212.707350] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7213.635992] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7218.605547] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7219.821329] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7221.539223] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 7221.543046] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 7226.848990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7226.853870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7226.860235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7226.862359] Lustre: Skipped 2 previous similar messages [ 7229.628279] Lustre: server umount lustre-MDT0000 complete [ 7231.207791] LustreError: 271499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176133 with bad export cookie 4634252967619235818 [ 7231.212570] LustreError: 271499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7231.213972] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7231.360986] Lustre: server umount lustre-MDT0001 complete [ 7242.711443] Lustre: server umount lustre-OST0000 complete [ 7254.898327] Lustre: server umount lustre-OST0001 complete [ 7261.637983] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7265.431239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7265.591895] LustreError: 276872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7265.614288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7265.617037] Lustre: Skipped 3 previous similar messages [ 7267.340666] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7270.621029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7272.353115] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7273.502151] Lustre: 278012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7276.333601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7278.596434] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7281.867766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7284.269433] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7290.351268] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 7290.352415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 7293.647912] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7295.567068] Lustre: 279887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7297.513979] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 7297.521621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 7300.995680] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 04:03:23 (1786176203) [ 7315.941462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7315.945839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7315.946086] Lustre: Skipped 5 previous similar messages [ 7315.950989] Lustre: Skipped 7 previous similar messages [ 7319.027380] Lustre: server umount lustre-MDT0000 complete [ 7320.509192] LustreError: 276854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176222 with bad export cookie 4634252967619242118 [ 7320.515617] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7320.518157] LustreError: 276854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7321.056828] LustreError: 276869:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7321.067743] LustreError: 276869:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7326.740854] Lustre: server umount lustre-MDT0001 complete [ 7334.536563] Lustre: server umount lustre-OST0000 complete [ 7342.194290] Lustre: server umount lustre-OST0001 complete [ 7345.051381] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 7348.775749] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7367.712317] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7371.713591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7371.822532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7371.833225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7371.873720] Lustre: lustre-MDT0000: new disk, initializing [ 7371.904155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7373.507733] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7378.625800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7378.672328] Lustre: 284115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7378.794265] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7380.202873] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7385.014666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7385.142349] Lustre: 285751:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7385.553624] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7387.091312] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7391.998318] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7393.247599] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7394.604776] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7399.767521] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7405.423391] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 04:05:07 (1786176307) [ 7408.611309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7408.613108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7408.616035] Lustre: Skipped 4 previous similar messages [ 7408.617868] Lustre: Skipped 4 previous similar messages [ 7412.294169] Lustre: server umount lustre-MDT0000 complete [ 7413.632219] LustreError: 284110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176315 with bad export cookie 4634252967619250000 [ 7413.637309] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7413.639822] LustreError: 284110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7413.729545] LustreError: 287147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7413.735621] LustreError: 287147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7413.775342] Lustre: server umount lustre-MDT0001 complete [ 7421.179810] Lustre: server umount lustre-OST0000 complete [ 7422.578804] Lustre: server umount lustre-OST0001 complete [ 7425.674400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7425.888032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7425.890445] Lustre: Skipped 7 previous similar messages [ 7427.420932] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7431.136856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7433.270737] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7436.292639] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7438.775959] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7441.088774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7443.312241] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7445.918412] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7458.151534] loop1: detected capacity change from 0 to 204800000 [ 7458.277540] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7458.281490] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7458.285565] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7460.092747] loop1: detected capacity change from 0 to 204800000 [ 7462.139129] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7462.185145] LustreError: 288850:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs998: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 7462.191788] LustreError: 288850:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs998 [ 7462.194274] LustreError: 288850:0:(mgs_handler.c:517:mgs_target_reg()) fs998-MDT0001: Error checking target index: rc = -13 [ 7462.197956] LustreError: fs998-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7462.201509] LustreError: 292580:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 7462.204463] LustreError: 292580:0:(tgt_mount.c:2129:server_put_super()) no obd fs998-MDT0001 [ 7462.206780] LustreError: 292580:0:(tgt_mount.c:116:server_deregister_mount()) fs998-MDT0001 not registered [ 7462.217654] Lustre: server umount fs998-MDT0001 complete [ 7462.219021] LustreError: 292580:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7465.417698] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7465.459396] Lustre: Setting parameter fs998-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs998-MDT0001 [ 7465.465082] Lustre: Skipped 2 previous similar messages [ 7465.488884] Lustre: srv-fs998-MDT0001: No data found on store. Initialize space. [ 7465.492151] Lustre: Skipped 4 previous similar messages [ 7465.540129] Lustre: fs998-MDT0001: new disk, initializing [ 7465.541824] Lustre: Skipped 3 previous similar messages [ 7465.595375] Lustre: srv-fs998-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7465.598533] Lustre: 293046:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs998-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7465.602073] LustreError: 293046:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs998-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7467.110388] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7470.496494] Lustre: server umount fs998-MDT0001 complete [ 7475.855302] loop1: detected capacity change from 0 to 204800000 [ 7477.930666] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7478.100571] Lustre: 294611:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs998-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7478.104219] LustreError: 294611:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs998-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7479.917802] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7481.889907] Lustre: server umount fs998-MDT0001 complete [ 7486.436422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7486.441084] Lustre: Skipped 6 previous similar messages [ 7491.609720] Lustre: server umount lustre-MDT0000 complete [ 7494.382450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7494.429728] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7495.816481] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7498.323282] loop1: detected capacity change from 0 to 204800000 [ 7499.756750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7500.213025] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7500.347698] Lustre: 297053:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs998-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7500.350537] LustreError: 297053:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs998-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7501.673831] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7503.303123] Lustre: server umount fs998-MDT0001 complete [ 7509.985015] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7524.319196] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7524.404515] Lustre: server umount lustre-OST0000 complete [ 7530.467077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7532.257952] Lustre: server umount lustre-MDT0000 complete [ 7533.462463] LustreError: 288842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176435 with bad export cookie 4634252967619252674 [ 7533.466714] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7533.467054] LustreError: 288842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7533.593629] Lustre: server umount lustre-MDT0001 complete [ 7537.339452] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7538.376591] Key type lgssc unregistered [ 7538.508469] LNet: 299395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7538.510973] LNetError: 299395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7538.520400] LNet: Removed LNI 192.168.202.116@tcp [ 7538.862106] Key type .llcrypt unregistered [ 7538.863346] Key type ._llcrypt unregistered [ 7546.659684] Key type ._llcrypt registered [ 7546.660848] Key type .llcrypt registered [ 7546.708239] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7547.137716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7547.206907] alg: No test for adler32 (adler32-zlib) [ 7548.065066] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 7548.153988] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 7549.727171] Key type lgssc registered [ 7550.063272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7552.502384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7552.508225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7553.654529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7554.982736] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7557.897475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7558.039862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7559.455738] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7561.686441] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7563.447477] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7565.706980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7565.846951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7567.767149] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7568.932157] LustreError: 302471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7568.940333] LustreError: 302471:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7570.216675] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7573.985996] LustreError: 302472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7578.081832] LustreError: 302471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7578.086262] LustreError: 302471:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7580.726571] LustreError: 303143:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7585.840285] LustreError: 302997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7585.844912] LustreError: 302997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7586.868905] loop1: detected capacity change from 0 to 204800000 [ 7586.932580] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7586.935797] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7586.939906] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7588.732600] loop1: detected capacity change from 0 to 204800000 [ 7590.633411] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7590.699627] Lustre: fs26761-OST0000: new disk, initializing [ 7590.702198] Lustre: srv-fs26761-OST0000: No data found on store. Initialize space. [ 7590.706381] Lustre: 304397:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7590.740129] Lustre: fs26761-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7592.601179] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7596.080052] LustreError: 302471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7596.085047] LustreError: 302471:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7596.218384] Lustre: server umount fs26761-OST0000 complete [ 7600.121245] loop1: detected capacity change from 0 to 204800000 [ 7601.939408] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7602.004789] Lustre: fs26761-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7603.875600] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7606.700346] loop2: detected capacity change from 0 to 204800000 [ 7606.794476] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7606.797653] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7606.801181] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7608.566460] loop2: detected capacity change from 0 to 204800000 [ 7610.404983] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7610.436880] LustreError: fs26761-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 7610.440781] LustreError: fs26761-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7610.443421] LustreError: 307433:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 7610.445534] LustreError: 307433:0:(tgt_mount.c:2129:server_put_super()) no obd fs26761-OST0001 [ 7610.447353] LustreError: 307433:0:(tgt_mount.c:116:server_deregister_mount()) fs26761-OST0001 not registered [ 7610.456853] Lustre: server umount fs26761-OST0001 complete [ 7610.458273] LustreError: 307433:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7613.848595] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7613.888386] Lustre: fs26761-OST0001: new disk, initializing [ 7613.890108] Lustre: srv-fs26761-OST0001: No data found on store. Initialize space. [ 7613.892596] Lustre: 307982:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7613.912556] Lustre: fs26761-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7613.921597] LustreError: 303143:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7613.925914] LustreError: 303143:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 7615.985075] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7619.567819] Lustre: server umount fs26761-OST0000 complete [ 7622.762533] Lustre: server umount fs26761-OST0001 complete [ 7627.072542] loop1: detected capacity change from 0 to 204800000 [ 7629.168663] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7631.236498] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7633.439522] loop2: detected capacity change from 0 to 204800000 [ 7635.257273] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7635.321623] Lustre: fs26761-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7635.323754] Lustre: Skipped 1 previous similar message [ 7637.196056] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7640.182741] Lustre: server umount fs26761-OST0000 complete [ 7643.388165] Lustre: server umount fs26761-OST0001 complete [ 7649.765126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7649.766605] LustreError: 302471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7649.767516] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7649.768920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7649.775076] LustreError: 302471:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 7649.778247] Lustre: Skipped 1 previous similar message [ 7653.139715] Lustre: server umount lustre-OST0000 complete [ 7654.880414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7654.881187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7654.885513] Lustre: Skipped 1 previous similar message [ 7654.887484] Lustre: Skipped 1 previous similar message [ 7660.000485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7660.002480] Lustre: Skipped 1 previous similar message [ 7660.638174] Lustre: server umount lustre-MDT0000 complete [ 7661.878411] LustreError: 302495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176563 with bad export cookie 13980210423488615471 [ 7661.881810] LustreError: 302495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7661.882458] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7662.005562] Lustre: server umount lustre-MDT0001 complete [ 7665.759351] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7666.769563] Key type lgssc unregistered [ 7666.896360] LNet: 313894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7666.898744] LNetError: 313894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7666.906372] LNet: Removed LNI 192.168.202.116@tcp [ 7667.213109] Key type .llcrypt unregistered [ 7667.214250] Key type ._llcrypt unregistered [ 7677.604456] Key type ._llcrypt registered [ 7677.605670] Key type .llcrypt registered [ 7677.645310] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 7684.341176] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7684.838712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7684.849660] alg: No test for adler32 (adler32-zlib) [ 7685.710829] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 7685.816313] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 7687.399160] Key type lgssc registered [ 7687.772769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7699.996623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7700.002044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7701.087915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7701.098193] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7701.135395] Lustre: lustre-MDT0000: new disk, initializing [ 7701.156120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7701.161889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7702.517730] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7706.657114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7706.705073] Lustre: 317740:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7706.757514] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7706.759646] Lustre: Skipped 1 previous similar message [ 7706.800666] Lustre: lustre-MDT0001: new disk, initializing [ 7706.832534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7706.844471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7706.849612] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7708.141805] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7711.861381] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7713.771307] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7716.024235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7716.141520] Lustre: lustre-OST0000: new disk, initializing [ 7716.144396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7716.148874] Lustre: 319707:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7716.177142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7718.097180] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7721.761150] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7722.499860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7722.503125] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7722.541761] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7727.602795] Lustre: Mounted lustre-client - version 2.17.56_50_gc905d6b [ 7727.939161] Lustre: Unmounted lustre-client [ 7732.707518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7732.707717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7732.711601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7732.713565] LustreError: Skipped 1 previous similar message [ 7732.717850] Lustre: Skipped 1 previous similar message [ 7737.113564] Lustre: server umount lustre-OST0000 complete [ 7742.946810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7742.947338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7742.948134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7742.956582] Lustre: Skipped 2 previous similar messages [ 7744.484043] Lustre: server umount lustre-MDT0000 complete [ 7745.705209] LustreError: 317733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176647 with bad export cookie 7825722714750237204 [ 7745.707785] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7745.708477] LustreError: 317733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7745.831416] Lustre: server umount lustre-MDT0001 complete [ 7749.620814] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7750.742639] Key type lgssc unregistered [ 7778.527361] LNet: 321870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7778.529679] LNetError: 321870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7853.285342] LNet: Removed LNI 192.168.202.116@tcp [ 7853.605957] Key type .llcrypt unregistered [ 7853.607216] Key type ._llcrypt unregistered [ 7859.529262] Key type ._llcrypt registered [ 7859.530323] Key type .llcrypt registered [ 7859.568160] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 7865.887706] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7866.225171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7866.262901] alg: No test for adler32 (adler32-zlib) [ 7867.119090] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 7867.202345] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 7868.775132] Key type lgssc registered [ 7869.139509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7879.145450] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 04:13:01 (1786176781) [ 7881.471051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7881.483329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7882.574720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7882.585952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7882.625173] Lustre: lustre-MDT0000: new disk, initializing [ 7882.648525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7882.655941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7884.099918] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7888.357603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7888.400846] Lustre: 325066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7888.429773] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7888.431622] Lustre: Skipped 1 previous similar message [ 7888.467824] Lustre: lustre-MDT0001: new disk, initializing [ 7888.506878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7888.519290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7888.522690] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7889.891890] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7893.591839] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7895.460934] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7897.704640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7897.829419] Lustre: lustre-OST0000: new disk, initializing [ 7897.831225] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7897.833784] Lustre: 327035:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7897.865701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7899.848429] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7901.238810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7901.242679] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7901.275177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7903.638635] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7911.378324] Lustre: server umount lustre-OST0000 complete [ 7914.464106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7914.467412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7914.470933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7916.512364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7916.512955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7921.632271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7921.634361] Lustre: Skipped 2 previous similar messages [ 7922.686537] Lustre: server umount lustre-MDT0000 complete [ 7923.960669] LustreError: 325864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176825 with bad export cookie 14453587422333597358 [ 7923.964579] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7923.964928] LustreError: 325864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7934.480960] Lustre: server umount lustre-MDT0001 complete [ 7938.408325] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7939.627204] Key type lgssc unregistered [ 7939.764580] LNet: 328944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7939.768803] LNetError: 328944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7939.776663] LNet: Removed LNI 192.168.202.116@tcp [ 7940.128498] Key type .llcrypt unregistered [ 7940.130394] Key type ._llcrypt unregistered [ 7948.375625] Key type ._llcrypt registered [ 7948.377411] Key type .llcrypt registered [ 7948.419746] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7948.778153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7948.858061] alg: No test for adler32 (adler32-zlib) [ 7949.708594] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 7949.797588] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 7951.375162] Key type lgssc registered [ 7951.771785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7954.485365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7954.491624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7955.642277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7957.165888] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7960.074527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7960.198937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7961.525713] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7963.935525] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7965.856609] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7968.218331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7968.350689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7970.256803] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7972.738666] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7986.549315] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7987.097217] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7991.265328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7991.267145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7991.267348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7991.270181] Lustre: Skipped 2 previous similar messages [ 7993.398323] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.16@tcp (stopping) [ 7993.400907] Lustre: Skipped 2 previous similar messages [ 7994.598816] Lustre: server umount lustre-MDT0000 complete [ 7995.949188] LustreError: 330176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176897 with bad export cookie 12534421560558852892 [ 7995.953082] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7995.954278] LustreError: 330176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7996.384071] LustreError: 330191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7996.384337] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7996.389154] LustreError: 330191:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7996.395789] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7998.510885] LustreError: 330190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7998.516920] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.16@tcp (stopping) [ 8000.416097] LustreError: 330191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8002.168400] Lustre: server umount lustre-MDT0001 complete [ 8003.637824] Lustre: lustre-OST0000: Not available for connect from 192.168.202.16@tcp (stopping) [ 8003.640139] Lustre: Skipped 1 previous similar message [ 8009.551817] Lustre: server umount lustre-OST0000 complete [ 8012.364397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8013.811288] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8014.794140] Lustre: 334393:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.116@tcp [ 8015.519564] Lustre: server umount MGS complete [ 8018.472855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8018.639518] LustreError: 334963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8018.666687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8020.043427] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8022.861809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8022.985542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8024.370328] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8027.238745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8029.348330] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8031.821981] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8035.401770] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 8037.230207] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8037.785657] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8042.979053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8042.980410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8042.983502] Lustre: Skipped 1 previous similar message [ 8042.985817] Lustre: Skipped 1 previous similar message [ 8042.988278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8045.009539] Lustre: server umount lustre-OST0000 complete [ 8048.097241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8048.100970] Lustre: Skipped 1 previous similar message [ 8052.377710] Lustre: server umount lustre-MDT0000 complete [ 8053.216112] LustreError: 335692:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8053.220580] LustreError: 335692:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8053.658614] LustreError: 334943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176955 with bad export cookie 12534421560558853718 [ 8053.661965] LustreError: 334943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8053.663301] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8053.799535] Lustre: server umount lustre-MDT0001 complete [ 8057.698698] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8058.851093] Key type lgssc unregistered [ 8059.008422] LNet: 338401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8059.010847] LNetError: 338401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8059.025449] LNet: Removed LNI 192.168.202.116@tcp [ 8059.376232] Key type .llcrypt unregistered [ 8059.377525] Key type ._llcrypt unregistered [ 8065.620333] Key type ._llcrypt registered [ 8065.621329] Key type .llcrypt registered [ 8065.664160] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 8072.381243] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8072.894941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8072.915844] alg: No test for adler32 (adler32-zlib) [ 8073.770310] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 8073.857676] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 8075.439182] Key type lgssc registered [ 8075.803695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8087.584796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8087.590237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8088.683271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8088.694978] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8088.739016] Lustre: lustre-MDT0000: new disk, initializing [ 8088.761513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8088.767625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8090.230388] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8094.404282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8094.452438] Lustre: 341533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8094.479541] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8094.481420] Lustre: Skipped 1 previous similar message [ 8094.524174] Lustre: lustre-MDT0001: new disk, initializing [ 8094.561474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8094.576265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8094.579732] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8095.980990] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8099.751822] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8101.701354] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8104.007203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8104.123164] Lustre: lustre-OST0000: new disk, initializing [ 8104.124909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8104.127344] Lustre: 343499:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8104.163140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8106.228022] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8106.625619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8106.628578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8106.666140] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8110.089514] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8111.613918] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8111.702362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8113.141791] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8113.222613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8117.217073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8117.218773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8117.219520] LustreError: Skipped 1 previous similar message [ 8117.220127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8117.224960] Lustre: Skipped 1 previous similar message [ 8122.323923] Lustre: server umount lustre-OST0000 complete [ 8125.407898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8125.410130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8125.414202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8125.416044] Lustre: Skipped 1 previous similar message [ 8127.456156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8127.456901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8129.821119] Lustre: server umount lustre-MDT0000 complete [ 8131.152121] LustreError: 341527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786177033 with bad export cookie 17496097016167137894 [ 8131.154517] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8131.156518] LustreError: 341527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8131.288751] Lustre: server umount lustre-MDT0001 complete [ 8134.407960] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 04:17:16 (1786177036) [ 8139.855419] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 8143.141398] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8155.661439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8155.755305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8155.765913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8155.809614] Lustre: lustre-MDT0000: new disk, initializing [ 8155.837686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8155.844737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8157.356449] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8161.752804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8161.796128] Lustre: 348248:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8161.820064] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8161.821787] Lustre: Skipped 1 previous similar message [ 8161.861650] Lustre: lustre-MDT0001: new disk, initializing [ 8161.895901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8161.911473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8161.914694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8163.357055] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8167.224586] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8169.245671] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8171.689886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8171.817745] Lustre: lustre-OST0000: new disk, initializing [ 8171.819732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8171.822460] Lustre: 350217:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8171.854242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8173.973319] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8177.655453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8177.660034] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8177.697346] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8177.820324] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8187.876338] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8187.877257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8187.878961] LustreError: Skipped 1 previous similar message [ 8187.884716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8187.888619] Lustre: Skipped 2 previous similar messages [ 8189.528160] Lustre: server umount lustre-OST0000 complete [ 8192.480017] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8192.483750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8192.488959] Lustre: Skipped 1 previous similar message [ 8192.490972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8203.231980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8203.233660] Lustre: Skipped 4 previous similar messages [ 8205.279165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8205.350919] Lustre: server umount lustre-MDT0000 complete [ 8206.623309] LustreError: 348239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786177108 with bad export cookie 17496097016167138762 [ 8206.627308] LustreError: 348239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8206.627601] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8206.759301] Lustre: server umount lustre-MDT0001 complete [ 8210.620348] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8211.722370] Key type lgssc unregistered [ 8211.854407] LNet: 352367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8211.858655] LNetError: 352367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8211.866490] LNet: Removed LNI 192.168.202.116@tcp [ 8212.227919] Key type .llcrypt unregistered [ 8212.229790] Key type ._llcrypt unregistered [ 8245.509636] Key type ._llcrypt registered [ 8245.510776] Key type .llcrypt registered [ 8245.552064] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8245.675631] Key type .llcrypt unregistered [ 8245.677138] Key type ._llcrypt unregistered [ 8273.638689] Key type ._llcrypt registered [ 8273.639562] Key type .llcrypt registered [ 8273.679794] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8273.808341] Key type .llcrypt unregistered [ 8273.809634] Key type ._llcrypt unregistered [ 8285.612428] Key type ._llcrypt registered [ 8285.613540] Key type .llcrypt registered [ 8285.653940] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8285.773324] Key type .llcrypt unregistered [ 8285.774561] Key type ._llcrypt unregistered [ 8297.212711] Key type ._llcrypt registered [ 8297.214612] Key type .llcrypt registered [ 8297.255848] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8297.368240] Key type .llcrypt unregistered [ 8297.369448] Key type ._llcrypt unregistered [ 8325.790838] Key type ._llcrypt registered [ 8325.792393] Key type .llcrypt registered [ 8325.834802] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8325.958329] Key type .llcrypt unregistered [ 8325.960267] Key type ._llcrypt unregistered [ 8337.651355] Key type ._llcrypt registered [ 8337.652385] Key type .llcrypt registered [ 8337.695894] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8337.812218] Key type .llcrypt unregistered [ 8337.813525] Key type ._llcrypt unregistered [ 8349.736431] Key type ._llcrypt registered [ 8349.737677] Key type .llcrypt registered [ 8349.782687] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8349.909279] Key type .llcrypt unregistered [ 8349.910486] Key type ._llcrypt unregistered [ 8361.489938] Key type ._llcrypt registered [ 8361.491780] Key type .llcrypt registered [ 8361.532053] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8361.683229] Key type .llcrypt unregistered [ 8361.684446] Key type ._llcrypt unregistered [ 8377.490452] Key type ._llcrypt registered [ 8377.491791] Key type .llcrypt registered [ 8377.540693] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8377.698378] Key type .llcrypt unregistered [ 8377.700415] Key type ._llcrypt unregistered [ 8392.663411] Key type ._llcrypt registered [ 8392.664650] Key type .llcrypt registered [ 8392.720991] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8392.876298] Key type .llcrypt unregistered [ 8392.878310] Key type ._llcrypt unregistered [ 8422.877937] Key type ._llcrypt registered [ 8422.878853] Key type .llcrypt registered [ 8422.915479] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8423.046526] Key type .llcrypt unregistered [ 8423.047392] Key type ._llcrypt unregistered [ 8440.826041] Key type ._llcrypt registered [ 8440.826984] Key type .llcrypt registered [ 8440.863859] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8440.976399] Key type .llcrypt unregistered [ 8440.977522] Key type ._llcrypt unregistered [ 8461.697444] Key type ._llcrypt registered [ 8461.699353] Key type .llcrypt registered [ 8461.746980] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8461.885505] Key type .llcrypt unregistered [ 8461.887465] Key type ._llcrypt unregistered [ 8526.432777] Key type ._llcrypt registered [ 8526.433953] Key type .llcrypt registered [ 8526.500816] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 8546.197908] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8546.866174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8546.963167] alg: No test for adler32 (adler32-zlib) [ 8548.218887] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 8548.600173] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 8550.367466] Key type lgssc registered [ 8551.302890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8589.664549] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8589.690154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8590.965248] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 8591.006072] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 8591.084477] Lustre: tst250fs-MDT0000: new disk, initializing [ 8591.189714] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8591.214756] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8595.446404] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8604.184249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8604.265896] Lustre: 372361:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8604.307754] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 8604.310557] Lustre: Skipped 1 previous similar message [ 8604.390375] Lustre: tst250fs-MDT0001: new disk, initializing [ 8604.446981] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8604.469702] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8604.484288] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8608.551826] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8613.445204] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8620.486824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8620.813467] Lustre: tst250fs-OST0000: new disk, initializing [ 8620.816467] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 8620.822195] Lustre: 373993:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8620.911763] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8626.780688] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8626.787758] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8626.903594] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8627.117062] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8638.506938] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8638.745977] Lustre: tst250fs-OST0001: new disk, initializing [ 8638.756671] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 8638.768718] Lustre: 374865:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8638.917472] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8644.891487] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8648.932418] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8648.945990] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8649.108191] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8650.746511] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 8652.929564] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [ 8652.934483] Lustre: Skipped 2 previous similar messages [ 8671.200833] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8671.220467] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8671.248519] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8674.806060] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8674.807874] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8676.927845] Lustre: server umount tst250fs-MDT0000 complete [ 8680.243587] LustreError: 373992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786177582 with bad export cookie 208337350177641125 [ 8680.248983] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8680.266275] LustreError: 373992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8680.664594] Lustre: server umount tst250fs-MDT0001 complete [ 8694.372247] Lustre: server umount tst250fs-OST0000 complete [ 8707.724581] Lustre: server umount tst250fs-OST0001 complete [ 8724.538397] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 8733.137516] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8765.140318] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 04:27:46 (1786177666) [ 8766.845900] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 8782.672605] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 8792.413364] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8824.994287] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8489 sec ========== 04:28:46 (1786177726) [ 8826.669260] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:28:47 (1786177727) === [ 8829.532663] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:28:50 (1786177730) === [ 8872.261804] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8875.973653] Key type lgssc unregistered [ 8876.220769] LNet: 383998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8876.241348] LNetError: 383998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8876.272848] LNet: Removed LNI 192.168.202.116@tcp [ 8877.328327] Key type .llcrypt unregistered [ 8877.331100] Key type ._llcrypt unregistered