[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 690866679 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002389] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.006860] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009134] LSM: Security Framework initializing [ 0.010053] Yama: becoming mindful. [ 0.011038] SELinux: Initializing. [ 0.012072] *** VALIDATE selinux *** [ 0.020817] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025063] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028116] *** VALIDATE tmpfs *** [ 0.030027] *** VALIDATE proc *** [ 0.031246] *** VALIDATE cgroup *** [ 0.032012] *** VALIDATE cgroup2 *** [ 0.033268] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036031] Spectre V2 : User space: Vulnerable [ 0.037010] Speculative Store Bypass: Vulnerable [ 0.040110] debug: unmapping init [mem 0xffffffff94859000-0xffffffff94860fff] [ 0.043279] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044687] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045025] ... version: 2 [ 0.046015] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048011] ... value mask: 0000ffffffffffff [ 0.049013] ... max period: 00007fffffffffff [ 0.050016] ... fixed-purpose events: 3 [ 0.051012] ... event mask: 000000070000000f [ 0.053221] rcu: Hierarchical SRCU implementation. [ 0.055405] smp: Bringing up secondary CPUs ... [ 0.056623] x86: Booting SMP configuration: [ 0.057027] .... node #0, CPUs: #1 #2 #3 [ 0.064082] smp: Brought up 1 node, 4 CPUs [ 0.066013] smpboot: Max logical packages: 1 [ 0.067013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.107168] node 0 deferred pages initialised in 38ms [ 0.111287] devtmpfs: initialized [ 0.113250] x86/mm: Memory block size: 128MB [ 0.116114] gcov: version magic: 0x41383552 [ 0.119220] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.123111] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.127420] pinctrl core: initialized pinctrl subsystem [ 0.129245] [ 0.131010] ************************************************************* [ 0.134013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.137013] ** ** [ 0.140059] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.142024] ** ** [ 0.145015] ** This means that this kernel is built to expose internal ** [ 0.147012] ** IOMMU data structures, which may compromise security on ** [ 0.151014] ** your system. ** [ 0.154012] ** ** [ 0.157010] ** If you see this message and you are not debugging the ** [ 0.160049] ** kernel, report this immediately to your vendor! ** [ 0.163012] ** ** [ 0.166014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170011] ************************************************************* [ 0.173595] NET: Registered protocol family 16 [ 0.176624] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.179088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.183091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.187133] cpuidle: using governor menu [ 0.189000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.193715] PCI: Using configuration type 1 for base access [ 0.203200] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.261240] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.266086] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.269039] cryptd: max_cpu_qlen set to 1000 [ 0.271251] ACPI: Added _OSI(Module Device) [ 0.273013] ACPI: Added _OSI(Processor Device) [ 0.274011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.275010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.280068] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.286397] ACPI: Interpreter enabled [ 0.288070] ACPI: PM: (supports S0 S3 S4 S5) [ 0.290019] ACPI: Using IOAPIC for interrupt routing [ 0.292285] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.297490] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.309632] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.313060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.316063] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.320084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.324302] acpiphp: Slot [2] registered [ 0.326133] acpiphp: Slot [5] registered [ 0.327104] acpiphp: Slot [6] registered [ 0.328095] acpiphp: Slot [7] registered [ 0.329159] acpiphp: Slot [8] registered [ 0.330102] acpiphp: Slot [9] registered [ 0.331109] acpiphp: Slot [10] registered [ 0.332089] acpiphp: Slot [3] registered [ 0.334066] acpiphp: Slot [4] registered [ 0.335085] acpiphp: Slot [11] registered [ 0.336104] acpiphp: Slot [12] registered [ 0.338141] acpiphp: Slot [13] registered [ 0.340079] acpiphp: Slot [14] registered [ 0.341162] acpiphp: Slot [15] registered [ 0.342099] acpiphp: Slot [16] registered [ 0.344106] acpiphp: Slot [17] registered [ 0.346132] acpiphp: Slot [18] registered [ 0.347112] acpiphp: Slot [19] registered [ 0.348097] acpiphp: Slot [20] registered [ 0.350121] acpiphp: Slot [21] registered [ 0.351088] acpiphp: Slot [22] registered [ 0.352087] acpiphp: Slot [23] registered [ 0.354081] acpiphp: Slot [24] registered [ 0.355084] acpiphp: Slot [25] registered [ 0.356080] acpiphp: Slot [26] registered [ 0.357085] acpiphp: Slot [27] registered [ 0.358097] acpiphp: Slot [28] registered [ 0.360075] acpiphp: Slot [29] registered [ 0.361077] acpiphp: Slot [30] registered [ 0.362188] acpiphp: Slot [31] registered [ 0.364142] PCI host bridge to bus 0000:00 [ 0.366028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.369037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.373038] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.375089] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.380033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.383034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.386203] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.390226] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.394076] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.402015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.406632] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.409025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.411015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.413021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.415463] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.418714] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.422050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.426008] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.430976] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.442016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.447017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.454425] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.473022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.488023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.528021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.546164] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.559016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.576033] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.608021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.624612] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.638019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.650019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.672020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.687936] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.700021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.709020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.732021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.755441] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.763019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.770017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.785026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.796056] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.805020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.815022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.842020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.854864] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.857289] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.859364] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.861299] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.863176] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.867147] iommu: Default domain type: Passthrough [ 0.869240] SCSI subsystem initialized [ 0.870118] ACPI: bus type USB registered [ 0.871085] usbcore: registered new interface driver usbfs [ 0.873060] usbcore: registered new interface driver hub [ 0.874061] usbcore: registered new device driver usb [ 0.875154] pps_core: LinuxPPS API ver. 1 registered [ 0.877012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.879107] PTP clock support registered [ 0.881073] EDAC MC: Ver: 3.0.0 [ 0.881963] PCI: Using ACPI for IRQ routing [ 0.885117] NetLabel: Initializing [ 0.886014] NetLabel: domain hash size = 128 [ 0.888025] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.890088] NetLabel: unlabeled traffic allowed by default [ 0.892229] vgaarb: loaded [ 0.893458] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.894010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.901863] clocksource: Switched to clocksource kvm-clock [ 1.024102] VFS: Disk quotas dquot_6.6.0 [ 1.027176] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.030636] *** VALIDATE ramfs *** [ 1.032155] *** VALIDATE hugetlbfs *** [ 1.034074] pnp: PnP ACPI init [ 1.038482] pnp: PnP ACPI: found 6 devices [ 1.057685] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.061230] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.063650] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.066070] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.068973] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.071940] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.075405] NET: Registered protocol family 2 [ 1.078893] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.085296] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.090154] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.095764] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.098447] TCP: Hash tables configured (established 65536 bind 65536) [ 1.100672] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.103032] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.105211] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.107544] NET: Registered protocol family 1 [ 1.109856] RPC: Registered named UNIX socket transport module. [ 1.111521] RPC: Registered udp transport module. [ 1.112840] RPC: Registered tcp transport module. [ 1.114095] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.115824] NET: Registered protocol family 44 [ 1.117037] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.118952] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.120816] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.122566] PCI: CLS 0 bytes, default 64 [ 1.123755] Unpacking initramfs... [ 2.554546] debug: unmapping init [mem 0xffff95aa7cc54000-0xffff95aa7ffbffff] [ 2.558738] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.561464] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.565242] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.142969] Initialise system trusted keyrings [ 3.145030] Key type blacklist registered [ 3.147110] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.157323] zbud: loaded [ 3.160485] *** VALIDATE nfs *** [ 3.162106] *** VALIDATE nfs4 *** [ 3.163854] pstore: using deflate compression [ 3.167960] Platform Keyring initialized [ 3.282681] NET: Registered protocol family 38 [ 3.284937] Key type asymmetric registered [ 3.286696] Asymmetric key parser 'x509' registered [ 3.289038] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.292345] io scheduler mq-deadline registered [ 3.294378] io scheduler kyber registered [ 3.296332] io scheduler bfq registered [ 3.298727] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.302511] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.306122] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.309756] ACPI: Power Button [PWRF] [ 3.315663] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.323133] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.342711] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.353396] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.375380] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.406919] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.437611] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.444267] Non-volatile memory driver v1.3 [ 3.446232] Linux agpgart interface v0.103 [ 3.478458] virtio_blk virtio1: [vda] 150080 512-byte logical blocks (76.8 MB/73.3 MiB) [ 3.481348] vda: detected capacity change from 0 to 76840960 [ 3.507895] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.512384] vdb: detected capacity change from 0 to 1073741824 [ 3.538857] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.542614] vdc: detected capacity change from 0 to 2621440000 [ 3.560854] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.563774] vdd: detected capacity change from 0 to 2621440000 [ 3.582466] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.585965] vde: detected capacity change from 0 to 4294967296 [ 3.601513] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.604612] vdf: detected capacity change from 0 to 4294967296 [ 3.617151] libphy: Fixed MDIO Bus: probed [ 3.623338] usbcore: registered new interface driver usbserial_generic [ 3.626450] usbserial: USB Serial support registered for generic [ 3.629514] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.634759] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.637330] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.640229] mousedev: PS/2 mouse device common for all mice [ 3.643254] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.645432] rtc_cmos 00:05: RTC can wake from S4 [ 3.653116] rtc_cmos 00:05: registered as rtc0 [ 3.657979] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.662729] intel_pstate: CPU model not supported [ 3.664367] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.670517] hid: raw HID events driver (C) Jiri Kosina [ 3.678360] usbcore: registered new interface driver usbhid [ 3.680619] usbhid: USB HID core driver [ 3.682576] drop_monitor: Initializing network drop monitor service [ 3.685532] Initializing XFRM netlink socket [ 3.687437] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.689596] NET: Registered protocol family 10 [ 3.692938] Segment Routing with IPv6 [ 3.698564] NET: Registered protocol family 17 [ 3.701450] mpls_gso: MPLS GSO support [ 3.708359] RAS: Correctable Errors collector initialized. [ 3.710917] AVX version of gcm_enc/dec engaged. [ 3.712836] AES CTR mode by8 optimization enabled [ 3.793992] sched_clock: Marking stable (3793920158, 0)->(4773405397, -979485239) [ 3.797807] registered taskstats version 1 [ 3.800327] Loading compiled-in X.509 certificates [ 3.802504] zswap: loaded using pool lzo/zbud [ 3.827051] Key type big_key registered [ 3.839992] Key type encrypted registered [ 3.841599] ima: No TPM chip found, activating TPM-bypass! [ 3.843657] ima: Allocated hash algorithm: sha1 [ 3.845552] ima: No architecture policies found [ 3.847443] evm: Initialising EVM extended attributes: [ 3.849426] evm: security.selinux [ 3.851151] evm: security.ima [ 3.852393] evm: security.capability [ 3.853883] evm: HMAC attrs: 0x1 [ 3.856415] rtc_cmos 00:05: setting system clock to 2026-09-07 03:43:28 UTC (1788752608) [ 3.864791] debug: unmapping init [mem 0xffffffff95803000-0xffffffff959fffff] [ 3.868352] debug: unmapping init [mem 0xffffffff94582000-0xffffffff94858fff] [ 3.877099] Write protecting the kernel read-only data: 28672k [ 3.881571] debug: unmapping init [mem 0xffffffff92c03000-0xffffffff92dfffff] [ 3.885373] debug: unmapping init [mem 0xffffffff93514000-0xffffffff935fffff] [ 3.921545] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.931616] systemd[1]: Detected virtualization kvm. [ 3.933923] systemd[1]: Detected architecture x86-64. [ 3.936123] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.964691] systemd[1]: No hostname configured. [ 3.966641] systemd[1]: Set hostname to . [ 3.969111] random: systemd: uninitialized urandom read (16 bytes read) [ 3.972263] systemd[1]: Initializing machine ID from random generator. [ 4.061112] random: ln: uninitialized urandom read (6 bytes read) [ 4.170397] random: systemd: uninitialized urandom read (16 bytes read) [ 4.173231] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.178404] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.184370] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Timers. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.225647] device-mapper: uevent: version 1.0.3 [ 5.227851] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.385122] random: fast init done [ 6.724535] virtio_net virtio0 ens2: renamed from eth0 [ 6.747165] scsi host0: ata_piix [ 6.754831] scsi host1: ata_piix [ 6.756500] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.759394] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.346202] dracut-initqueue[588]: RTNETLINK answers: File exists [ 11.478622] random: crng init done [ 11.480711] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ 12.805507] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.182502] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.506085] SELinux: Disabled at runtime. [ 14.570642] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.580286] systemd[1]: Detected virtualization kvm. [ 14.581869] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.475912] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.479752] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.489496] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.494509] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.498485] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.506609] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.517336] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ 15.662360] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 16.418490] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 16.762279] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 16.903027] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.957656] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.008075] EDAC sbridge: Ver: 1.1.2 [ 19.575283] Key type dns_resolver registered [ 19.921726] NFS: Registering the id_resolver key type [ 19.924161] Key type id_resolver registered [ 19.926291] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. Starting Hostname Service... [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg353-server login: [ 47.101749] libcfs: loading out-of-tree module taints kernel. [ 47.122779] Key type ._llcrypt registered [ 47.124871] Key type .llcrypt registered [ 47.174684] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 59.530463] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 61.245164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 61.277535] alg: No test for adler32 (adler32-zlib) [ 62.777462] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 63.997571] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 65.879639] Key type lgssc registered [ 67.574989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 83.066007] hrtimer: interrupt took 4996506 ns [ 85.034597] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 119.590780] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 130.592862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 130.627699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 131.868339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 131.910990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 132.029141] Lustre: lustre-MDT0000: new disk, initializing [ 132.101167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 132.133051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 136.749641] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 149.245799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 149.332237] Lustre: 6487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 149.356545] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 149.362064] Lustre: Skipped 1 previous similar message [ 149.480880] Lustre: lustre-MDT0001: new disk, initializing [ 149.557621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 149.600567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 149.611578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 153.855461] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 158.156812] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 166.452045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 166.780811] Lustre: lustre-OST0000: new disk, initializing [ 166.788168] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 166.802347] Lustre: 8428:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 166.887283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.507401] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 176.183147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 176.192857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 176.283426] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 185.567621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 185.716696] Lustre: lustre-OST0001: new disk, initializing [ 185.720477] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 185.728047] Lustre: 9498:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 185.803900] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 185.994027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 186.022804] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 186.108291] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 191.278620] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.531033] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 206.473847] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 211.425843] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing check_logdir /tmp/testlogs/ [ 215.264778] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing yml_node [ 219.733478] Lustre: DEBUG MARKER: Client: 2.17.58.43 [ 221.825629] Lustre: DEBUG MARKER: MDS: 2.17.58.43 [ 224.278436] Lustre: DEBUG MARKER: OSS: 2.17.58.43 [ 225.675524] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 6 23:47:09 EDT 2026 [ 240.000568] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 241.157836] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 257.002213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 257.015298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 257.046449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 258.015853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 258.016484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 258.043838] Lustre: Skipped 1 previous similar message [ 263.135703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 263.145992] Lustre: Skipped 6 previous similar messages [ 268.257836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 268.270100] Lustre: Skipped 3 previous similar messages [ 270.303211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 270.709878] Lustre: server umount lustre-MDT0000 complete [ 274.173751] LustreError: 8426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752878 with bad export cookie 8101043997808023692 [ 274.176180] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 274.186787] LustreError: 8426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 278.513877] LustreError: 6493:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 278.514307] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 278.526198] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 278.528373] LustreError: 6493:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 6 previous similar messages [ 278.558774] Lustre: Skipped 2 previous similar messages [ 279.520219] LustreError: 11209:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 279.556725] LustreError: 11209:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 280.657729] Lustre: server umount lustre-MDT0001 complete [ 284.615465] Lustre: server umount lustre-OST0000 complete [ 288.643148] Lustre: server umount lustre-OST0001 complete [ 293.956344] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 301.721355] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 331.960733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 332.133296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 332.154798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 332.250218] Lustre: lustre-MDT0000: new disk, initializing [ 332.339348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 332.353727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 336.100296] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 344.488849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 344.589299] Lustre: 15044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 344.632793] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 344.637773] Lustre: Skipped 1 previous similar message [ 344.702916] Lustre: lustre-MDT0001: new disk, initializing [ 344.787471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 344.824931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 344.836703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 348.458840] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 356.503678] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 361.044311] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 366.948590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 367.215581] Lustre: lustre-OST0000: new disk, initializing [ 367.226812] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 367.232155] Lustre: 17019:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 367.329695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 368.885238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 368.900371] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 369.047637] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 372.967229] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 381.851179] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 386.215735] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 386.463065] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 390.370858] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 390.593095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 394.720463] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 394.726761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 394.735330] LustreError: Skipped 1 previous similar message [ 394.741152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 394.757182] Lustre: Skipped 1 previous similar message [ 394.758345] Lustre: Skipped 3 previous similar messages [ 398.607584] Lustre: server umount lustre-OST0000 complete [ 404.966561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 404.985729] Lustre: Skipped 1 previous similar message [ 408.788471] Lustre: server umount lustre-MDT0000 complete [ 410.080979] LustreError: 15058:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 410.095352] LustreError: 15058:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 412.049352] LustreError: 15037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753016 with bad export cookie 8101043997808025225 [ 412.054125] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 412.066663] LustreError: 15037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 412.410966] Lustre: server umount lustre-MDT0001 complete [ 416.757664] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 23:50:20 (1788753020) [ 430.667971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 431.033625] LustreError: 19890:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 431.099792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 434.404959] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 436.195269] LustreError: 19891:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 441.325768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 445.163814] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 450.820679] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 455.561064] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 457.289940] Lustre: Failing over lustre-MDT0000 [ 457.572295] Lustre: server umount lustre-MDT0000 complete [ 461.280938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 461.284861] LustreError: 19886:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 461.300703] Lustre: Skipped 1 previous similar message [ 461.341429] LustreError: 19886:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 474.779673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 474.882378] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 478.866159] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 480.227672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 480.229893] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 480.260566] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 487.282981] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 490.762356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 490.782903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 490.791258] Lustre: Skipped 3 previous similar messages [ 495.585550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 505.311565] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 505.463080] Lustre: server umount lustre-MDT0000 complete [ 506.341696] LustreError: 19890:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.365530] LustreError: 19890:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [ 508.821024] LustreError: 22551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753113 with bad export cookie 8101043997808026639 [ 508.832162] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 509.238835] Lustre: server umount lustre-MDT0001 complete [ 515.855749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 516.255341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 516.263595] Lustre: Skipped 2 previous similar messages [ 519.678740] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 526.277501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 529.889269] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 534.537490] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 538.120432] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 539.726276] Lustre: Failing over lustre-MDT0000 [ 539.734365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.737093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 539.741984] Lustre: Skipped 2 previous similar messages [ 539.751059] Lustre: Skipped 6 previous similar messages [ 539.895611] Lustre: server umount lustre-MDT0000 complete [ 544.736980] LustreError: 23957:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.751862] LustreError: 23957:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 556.127989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 556.207224] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 560.165801] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 561.640942] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 561.647597] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 561.661794] Lustre: Skipped 1 previous similar message [ 561.671884] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 567.638216] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 571.881391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 577.345277] Lustre: server umount lustre-MDT0000 complete [ 581.002122] LustreError: 23210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753185 with bad export cookie 8101043997808027577 [ 581.023381] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 581.045398] LustreError: 23210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 581.639826] Lustre: server umount lustre-MDT0001 complete [ 597.352604] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 23:53:19 (1788753199) [ 615.267826] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 625.914992] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 663.972553] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 674.385957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 674.622609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 674.666572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 674.746042] Lustre: lustre-MDT0000: new disk, initializing [ 674.831227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 674.835630] Lustre: Skipped 2 previous similar messages [ 674.862517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 679.214542] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 688.965679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 689.092913] Lustre: 29603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 689.326721] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 693.851827] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 702.234874] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 707.930567] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 713.925437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 714.205703] Lustre: 31574:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 715.919377] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 715.954775] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 716.032349] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 716.058526] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 719.629147] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 728.134936] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 735.170104] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 752.105984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 752.108457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 752.115949] LustreError: Skipped 1 previous similar message [ 752.117929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 752.128495] Lustre: Skipped 3 previous similar messages [ 752.143654] Lustre: Skipped 5 previous similar messages [ 756.001209] Lustre: server umount lustre-OST0000 complete [ 761.319967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 765.893412] Lustre: server umount lustre-MDT0000 complete [ 767.469790] LustreError: 29616:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 767.485713] LustreError: 29616:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 7 previous similar messages [ 769.122498] LustreError: 31573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753373 with bad export cookie 8101043997808027983 [ 769.132118] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 769.145079] LustreError: 31573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 769.637903] Lustre: server umount lustre-MDT0001 complete [ 780.048152] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 782.633996] Key type lgssc unregistered [ 782.961655] LNet: 33549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.970493] LNetError: 33549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 782.984152] LNet: Removed LNI 192.168.203.153@tcp [ 783.930604] Key type .llcrypt unregistered [ 783.933600] Key type ._llcrypt unregistered [ 807.278560] Key type ._llcrypt registered [ 807.281898] Key type .llcrypt registered [ 807.378839] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 808.451072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 808.488382] alg: No test for adler32 (adler32-zlib) [ 809.482217] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 809.676817] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 811.335225] Key type lgssc registered [ 812.350723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 819.810489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 819.834906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 821.304905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 825.752137] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 830.440495] LustreError: 34961:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 830.459264] LustreError: 34961:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 833.524920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 833.927183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 838.447672] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 845.383938] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 850.742796] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 857.545310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 857.897955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 858.959442] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 864.682768] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 872.112247] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 883.401621] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 23:58:06 (1788753486) [ 884.942372] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 886.412823] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 892.461128] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 23:58:15 (1788753495) [ 900.135408] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 23:58:23 (1788753503) [ 901.031559] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 908.783234] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 908.786939] Lustre: Skipped 2 previous similar messages [ 910.606325] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 23:58:33 (1788753513) [ 913.738275] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 913.750942] Lustre: Skipped 3 previous similar messages [ 928.990970] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 928.997292] Lustre: Skipped 6 previous similar messages [ 931.256311] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 23:58:54 (1788753534) [ 933.644668] Lustre: *** cfs_fail_loc=131b, val=2*** [ 937.019937] Lustre: *** cfs_fail_loc=131b, val=2*** [ 944.606376] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 23:59:07 (1788753547) [ 946.390514] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 946.397328] Lustre: Skipped 3 previous similar messages [ 957.075613] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 23:59:20 (1788753560) [ 971.744801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 971.748077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 971.770317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 973.791931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 973.804324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 973.814839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 973.848085] Lustre: Skipped 1 previous similar message [ 976.040661] Lustre: server umount lustre-OST0000 complete [ 982.505077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 982.520804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 982.548804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 986.626959] Lustre: server umount lustre-MDT0000 complete [ 989.159578] LustreError: 34957:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 989.186359] LustreError: 34957:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 991.058405] LustreError: 34940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753595 with bad export cookie 13128285056650351534 [ 991.060426] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 991.070266] LustreError: 34940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 991.424903] Lustre: server umount lustre-MDT0001 complete [ 1003.378304] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 1006.168883] Key type lgssc unregistered [ 1006.518986] LNet: 42812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.534801] LNetError: 42812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.595777] LNet: Removed LNI 192.168.203.153@tcp [ 1008.351453] Key type .llcrypt unregistered [ 1008.353410] Key type ._llcrypt unregistered [ 1024.134509] Key type ._llcrypt registered [ 1024.136968] Key type .llcrypt registered [ 1024.230202] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 1038.049474] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1039.367807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1039.383609] alg: No test for adler32 (adler32-zlib) [ 1040.409417] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 1040.588603] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 1042.255208] Key type lgssc registered [ 1043.515934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1077.400202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1077.432620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1078.815147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1078.849430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1078.957775] Lustre: lustre-MDT0000: new disk, initializing [ 1079.071040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1079.115106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1084.023883] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1093.935529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1094.077326] Lustre: 45974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1094.176450] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1094.180563] Lustre: Skipped 1 previous similar message [ 1094.270737] Lustre: lustre-MDT0001: new disk, initializing [ 1094.429391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1094.530210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1094.560537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1099.431375] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1110.435132] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1116.247844] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1122.775921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1123.008340] Lustre: lustre-OST0000: new disk, initializing [ 1123.011148] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1123.017230] Lustre: 47944:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1123.106055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1128.836942] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1130.707073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1130.741335] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1130.874831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1138.852582] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1142.396409] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1142.587768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1146.293938] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1146.447664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1151.463786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1151.464334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1151.487771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1151.489197] LustreError: Skipped 1 previous similar message [ 1151.502076] Lustre: Skipped 1 previous similar message [ 1156.588633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1156.598432] Lustre: Skipped 1 previous similar message [ 1161.696675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1161.706532] Lustre: Skipped 1 previous similar message [ 1162.207193] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1162.334729] Lustre: server umount lustre-OST0000 complete [ 1166.304286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1166.313144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1166.325461] Lustre: Skipped 1 previous similar message [ 1166.330887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1171.659187] Lustre: server umount lustre-MDT0000 complete [ 1171.938647] LustreError: 45980:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1171.954056] LustreError: 45980:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 1174.664693] LustreError: 46770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753779 with bad export cookie 13744915830289203141 [ 1174.675197] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1174.680765] LustreError: 46770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1174.970797] Lustre: server umount lustre-MDT0001 complete [ 1182.701531] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 00:03:06 (1788753786) [ 1189.335554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1189.781739] LustreError: 50404:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1189.852458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1194.588617] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1194.978564] LustreError: 50405:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1200.098115] LustreError: 50404:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1202.669996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1202.944545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1207.067443] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1213.808856] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1219.161133] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1226.129870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1226.541335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1232.828841] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1239.925269] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1244.039651] LustreError: 52932:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1244.907717] Lustre: Modifying parameter general.timeout=1 in log params [ 1246.556226] Lustre: Modifying parameter general.timeout=3 in log params [ 1246.565384] Lustre: Skipped 1 previous similar message [ 1249.101230] Lustre: Modifying parameter general.timeout=6 in log params [ 1249.105936] Lustre: Skipped 2 previous similar messages [ 1253.678213] Lustre: Modifying parameter general.timeout=12 in log params [ 1253.682875] Lustre: Skipped 5 previous similar messages [ 1262.267351] Lustre: Modifying parameter general.timeout=24 in log params [ 1262.273376] Lustre: Skipped 11 previous similar messages [ 1278.566588] Lustre: Modifying parameter general.timeout=45 in log params [ 1278.572566] Lustre: Skipped 20 previous similar messages [ 1310.649285] Lustre: Modifying parameter general.timeout=86 in log params [ 1310.654504] Lustre: Skipped 40 previous similar messages [ 1351.523354] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 00:05:55 (1788753955) [ 1353.711493] Lustre: 59860:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1355.374884] Lustre: 59958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1358.720731] Lustre: 60172:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1358.729285] Lustre: 60172:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1375.280117] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 in log params [ 1375.286328] Lustre: Skipped 61 previous similar messages [ 1384.069885] Lustre: 61721:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1384.078728] Lustre: 61721:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1384.093783] LustreError: 61721:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1387.358085] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 00:06:30 (1788753990) [ 1397.728412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1397.736441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1397.754667] Lustre: Skipped 1 previous similar message [ 1397.759470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1397.765973] Lustre: Skipped 1 previous similar message [ 1399.781592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1399.797695] Lustre: Skipped 1 previous similar message [ 1403.099340] Lustre: server umount lustre-MDT0000 complete [ 1404.899657] LustreError: 50400:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1404.914267] LustreError: 50400:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1406.044721] LustreError: 51139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754010 with bad export cookie 13744915830289204016 [ 1406.059630] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1406.062040] LustreError: 51139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1406.432822] Lustre: server umount lustre-MDT0001 complete [ 1420.431735] Lustre: server umount lustre-OST0000 complete [ 1444.094962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1444.416458] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1444.577313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1448.865834] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1455.542800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1455.627711] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1455.647635] Lustre: 64251:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1459.039751] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1464.214556] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1468.669542] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1473.965611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1474.125473] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1479.834918] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1486.960570] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1488.396377] Lustre: 66815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1496.033406] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1496.037047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1496.051247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1496.056078] Lustre: Skipped 3 previous similar messages [ 1501.019168] Lustre: server umount lustre-OST0000 complete [ 1506.787030] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1506.810312] Lustre: Skipped 2 previous similar messages [ 1510.546956] Lustre: server umount lustre-MDT0000 complete [ 1511.904497] LustreError: 64256:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1511.923247] LustreError: 64256:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1513.922727] LustreError: 64243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754118 with bad export cookie 13744915830289206837 [ 1513.936694] LustreError: 64243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1513.938114] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1514.254075] Lustre: server umount lustre-MDT0001 complete [ 1523.329727] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 1525.841519] Key type lgssc unregistered [ 1526.099106] LNet: 68026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.105930] LNetError: 68026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1526.123057] LNet: Removed LNI 192.168.203.153@tcp [ 1526.777351] Key type .llcrypt unregistered [ 1526.781707] Key type ._llcrypt unregistered [ 1546.031371] Key type ._llcrypt registered [ 1546.034588] Key type .llcrypt registered [ 1546.146822] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1547.148610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1547.234988] alg: No test for adler32 (adler32-zlib) [ 1548.350273] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 1548.600969] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 1550.263214] Key type lgssc registered [ 1551.314640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1558.145377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1558.182359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1559.730285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1563.395777] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1570.273850] LustreError: 69416:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1570.296941] LustreError: 69416:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1570.312339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1570.744351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1574.537197] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1580.491478] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1584.824361] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1589.712417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1590.191382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1595.494710] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1603.257780] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1611.723034] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1621.992182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1622.004471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1622.031148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1625.063089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1625.063568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1625.068182] Lustre: Skipped 1 previous similar message [ 1625.075791] Lustre: Skipped 1 previous similar message [ 1626.597263] Lustre: server umount lustre-MDT0000 complete [ 1630.178169] LustreError: 69417:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1630.193528] LustreError: 69417:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1630.322717] LustreError: 69397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754234 with bad export cookie 136616157815960747 [ 1630.327881] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1630.345057] LustreError: 69397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1630.720859] Lustre: server umount lustre-MDT0001 complete [ 1645.407052] Lustre: server umount lustre-OST0000 complete [ 1670.944299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1671.332724] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1671.384344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1671.657277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1676.215494] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1684.446787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1684.521170] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1684.545894] Lustre: 74287:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1684.923883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1689.379271] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1696.715740] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1702.875383] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1709.578846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1709.782700] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1710.304831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1716.037974] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1721.572568] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1722.934306] Lustre: 76861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1724.401563] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1729.504169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1729.520592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1729.547896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1729.558687] Lustre: Skipped 1 previous similar message [ 1734.704909] Lustre: server umount lustre-OST0000 complete [ 1740.262117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1740.284689] Lustre: Skipped 1 previous similar message [ 1740.301361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1740.311844] Lustre: Skipped 3 previous similar messages [ 1744.289668] Lustre: server umount lustre-MDT0000 complete [ 1745.379354] LustreError: 74296:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1745.399952] LustreError: 74296:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1747.384195] LustreError: 74280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754352 with bad export cookie 136616157815961573 [ 1747.391318] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1747.402850] LustreError: 74280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1747.796792] Lustre: server umount lustre-MDT0001 complete [ 1757.480603] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 1760.305848] Key type lgssc unregistered [ 1760.547983] LNet: 78072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1760.564600] LNetError: 78072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1760.595213] LNet: Removed LNI 192.168.203.153@tcp [ 1761.494626] Key type .llcrypt unregistered [ 1761.496839] Key type ._llcrypt unregistered [ 1784.273617] Key type ._llcrypt registered [ 1784.278273] Key type .llcrypt registered [ 1784.445507] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1785.846897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1785.923945] alg: No test for adler32 (adler32-zlib) [ 1787.090688] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 1787.391302] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 1789.159211] Key type lgssc registered [ 1790.902645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1797.435436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1797.452882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1799.102444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1803.667550] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1809.377240] LustreError: 79481:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1809.386489] LustreError: 79481:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1810.667897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.309856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1815.611338] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1821.382843] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1825.687979] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1831.609775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1832.054394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1838.412593] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1845.557672] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1851.953745] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1852.729319] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1854.411855] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1854.419416] Lustre: Skipped 1 previous similar message [ 1856.835436] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1856.841714] Lustre: Skipped 2 previous similar messages [ 1860.867554] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1860.875937] Lustre: Skipped 4 previous similar messages [ 1869.443700] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1869.448788] Lustre: Skipped 10 previous similar messages [ 1885.924843] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1885.928900] Lustre: Skipped 21 previous similar messages [ 1918.634414] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1918.642257] Lustre: Skipped 41 previous similar messages [ 1938.939194] Lustre: 87574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1941.214678] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 00:15:44 (1788754544) [ 1942.725197] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1944.176392] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 00:15:47 (1788754547) [ 1948.134184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.147420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1948.173087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1953.264602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1953.277581] Lustre: Skipped 2 previous similar messages [ 1954.109418] Lustre: server umount lustre-OST0000 complete [ 1958.369706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1958.381843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1958.389076] Lustre: Skipped 2 previous similar messages [ 1958.405313] Lustre: Skipped 1 previous similar message [ 1963.489262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1963.497902] Lustre: Skipped 1 previous similar message [ 1963.887585] Lustre: server umount lustre-MDT0000 complete [ 1967.357637] LustreError: 82464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754571 with bad export cookie 2730521633092512964 [ 1967.363540] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1967.372032] LustreError: 82464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1967.912036] Lustre: server umount lustre-MDT0001 complete [ 1978.200519] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 1980.746731] Key type lgssc unregistered [ 1981.049951] LNet: 88831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1981.061499] LNetError: 88831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1981.082296] LNet: Removed LNI 192.168.203.153@tcp [ 1982.063248] Key type .llcrypt unregistered [ 1982.067840] Key type ._llcrypt unregistered [ 1986.304609] Key type ._llcrypt registered [ 1986.310131] Key type .llcrypt registered [ 1986.438841] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1990.245418] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules [ 1990.967460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1990.986420] alg: No test for adler32 (adler32-zlib) [ 1991.967994] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 1992.131442] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 1993.791222] Key type lgssc registered [ 1994.804678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1995.700658] LustreError: 90028:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2011.540487] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2017.757204] LustreError: 90028:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2019.654168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2019.672295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2021.044954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2025.065228] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2030.561853] LustreError: 91052:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2030.585423] LustreError: 91052:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2032.421208] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 00:17:15 (1788754635) [ 2035.684858] LustreError: 91053:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2036.244367] LustreError: 91084:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2036.252736] LustreError: 91084:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2036.258324] LustreError: 91084:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 2036.528855] Lustre: server umount lustre-MDT0000 complete [ 2048.067908] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 2050.407767] Key type lgssc unregistered [ 2050.705958] LNet: 92576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2050.715812] LNetError: 92576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2050.729168] LNet: Removed LNI 192.168.203.153@tcp [ 2051.498799] Key type .llcrypt unregistered [ 2051.502924] Key type ._llcrypt unregistered [ 2067.009835] Key type ._llcrypt registered [ 2067.012956] Key type .llcrypt registered [ 2067.145138] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2068.081378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2068.270318] alg: No test for adler32 (adler32-zlib) [ 2069.314329] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 2069.563578] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 2071.279180] Key type lgssc registered [ 2072.657471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2080.771926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2080.819723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2082.334182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2086.417707] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2091.490877] LustreError: 93789:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2091.512655] LustreError: 93789:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2095.541323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2096.327351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2102.224637] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2110.147753] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2115.838638] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2122.080078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2122.599586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2128.114804] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2135.734717] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2181.431963] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 00:19:44 (1788754784) [ 2182.808902] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2184.381519] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 00:19:47 (1788754787) [ 2188.258245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2188.266290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.285058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2192.356615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2192.357487] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2192.364487] Lustre: Skipped 1 previous similar message [ 2192.377346] Lustre: Skipped 1 previous similar message [ 2194.155565] Lustre: server umount lustre-MDT0000 complete [ 2197.473395] LustreError: 93785:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2197.520146] LustreError: 93785:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2197.607137] LustreError: 93771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754802 with bad export cookie 7071538328107075334 [ 2197.615474] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2197.620249] LustreError: 93771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2198.005784] Lustre: server umount lustre-MDT0001 complete [ 2212.168951] Lustre: server umount lustre-OST0000 complete [ 2223.116223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2223.602560] LustreError: 98064:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2223.765160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2228.034163] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2229.219022] LustreError: 98065:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2234.341402] LustreError: 98064:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2235.736286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2236.391767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2240.330177] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2247.351441] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2251.951755] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2264.004211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2264.181470] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2264.186951] LustreError: 98053:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2264.206821] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2264.216402] LustreError: 100147:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2264.221807] LustreError: 100147:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2264.225820] LustreError: 100147:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2264.272266] Lustre: server umount lustre-OST0000 complete [ 2264.274548] LustreError: 100147:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2271.528872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2271.799437] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2271.816049] LustreError: 98052:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2271.831482] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2271.843057] LustreError: 100611:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2271.848932] LustreError: 100611:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2271.859105] LustreError: 100611:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2271.953039] Lustre: server umount lustre-OST0000 complete [ 2271.962140] LustreError: 100611:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2280.372549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2280.438852] LustreError: 101133:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2280.520616] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2280.523980] Lustre: Found index 0 for lustre-OST0000, updating log [ 2280.528976] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2280.590822] Lustre: lustre-OST0000: new disk, initializing [ 2280.594226] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2280.602883] Lustre: 101148:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2280.827363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2287.052254] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2290.901855] Lustre: Failing over lustre-OST0000 [ 2291.162531] Lustre: server umount lustre-OST0000 complete [ 2297.826779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2297.836977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2297.855424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2297.863889] Lustre: Skipped 1 previous similar message [ 2301.329496] Lustre: server umount lustre-MDT0000 complete [ 2304.487874] LustreError: 98059:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2304.505711] LustreError: 98059:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2304.631116] LustreError: 98046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754909 with bad export cookie 7071538328107078085 [ 2304.633472] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2304.641993] LustreError: 98046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2304.651849] LustreError: 98795:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 2304.671770] LustreError: 98795:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 2315.431733] Lustre: server umount lustre-MDT0001 complete [ 2324.079195] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 00:22:07 (1788754927) [ 2341.985726] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2353.238645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.723929] LustreError: 103492:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2353.895773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2359.192788] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2367.818801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2373.588447] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2376.930348] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2383.277783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2383.865291] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2389.276451] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2393.063953] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2393.088151] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2393.103482] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2398.560414] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2398.692995] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2398.778931] Lustre: lustre-OST0001: new disk, initializing [ 2398.783666] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2398.790857] Lustre: 105811:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2399.107097] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2399.113708] Lustre: Skipped 2 previous similar messages [ 2405.003740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2405.019397] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2405.145873] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2406.017936] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2413.567812] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2420.020102] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2424.297414] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.53@tcp (stopping) [ 2424.312265] Lustre: Skipped 2 previous similar messages [ 2424.810074] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2424.820241] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.833797] Lustre: Skipped 1 previous similar message [ 2429.375203] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.53@tcp (stopping) [ 2429.380725] Lustre: Skipped 4 previous similar messages [ 2430.337575] Lustre: server umount lustre-MDT0001 complete [ 2430.954376] LustreError: 104224:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2430.969048] LustreError: 104224:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 2439.965662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2440.083117] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2440.091239] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2440.137606] Lustre: 103479:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2441.738800] LustreError: 107379:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2441.749406] LustreError: 107379:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2441.764267] LustreError: 103484:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2441.909428] LustreError: 107392:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.153@tcp: cfg command failed: rc = -17 [ 2441.928423] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2441.928423] [ 2445.078708] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2446.316550] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2455.126276] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 00:24:18 (1788755058) [ 2473.955460] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2491.550679] Lustre: 109425:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2513.843381] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2517.218068] Lustre: 110560:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2594.273035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2594.284322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2594.302609] Lustre: Skipped 2 previous similar messages [ 2594.307903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2599.792710] Lustre: server umount lustre-MDT0000 complete [ 2599.904248] LustreError: 110744:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2599.927775] LustreError: 110744:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 9 previous similar messages [ 2602.920942] LustreError: 106791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755207 with bad export cookie 7071538328107079086 [ 2602.923966] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2602.930039] LustreError: 106791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2603.173854] Lustre: server umount lustre-MDT0001 complete [ 2616.886563] Lustre: server umount lustre-OST0000 complete [ 2630.307167] Lustre: server umount lustre-OST0001 complete [ 2634.554340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2642.866478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2652.087287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2661.146932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2679.302828] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2689.674654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2689.713396] Lustre: lustre-MDT0000: reset Object Index mappings [ 2690.306052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2690.318935] Lustre: Skipped 1 previous similar message [ 2695.118607] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2702.776656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2702.797522] Lustre: lustre-MDT0001: reset Object Index mappings [ 2708.121885] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2710.574647] Lustre: 115661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2716.803644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2723.631595] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2727.605276] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2732.430950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2735.028574] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2735.030859] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2735.037085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2738.446074] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2745.912796] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2749.569968] Lustre: 117539:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2799.521069] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 00:30:02 (1788755402) [ 2826.208677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2826.217959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2826.233930] Lustre: Skipped 3 previous similar messages [ 2826.243972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2826.249172] Lustre: Skipped 3 previous similar messages [ 2832.167318] Lustre: server umount lustre-MDT0000 complete [ 2835.024426] LustreError: 114495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755439 with bad export cookie 7071538328107125937 [ 2835.030964] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2835.039217] LustreError: 114495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2841.487320] Lustre: server umount lustre-MDT0001 complete [ 2850.737716] Lustre: server umount lustre-OST0000 complete [ 2859.487961] Lustre: server umount lustre-OST0001 complete [ 2864.261467] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 2873.905680] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2907.718391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2907.938954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2908.006804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2908.153606] Lustre: lustre-MDT0000: new disk, initializing [ 2908.237754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2908.246763] Lustre: Skipped 3 previous similar messages [ 2908.274696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2913.297198] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2920.428474] Lustre: server umount lustre-MDT0000 complete [ 2930.846882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2931.183953] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2931.195776] Lustre: 121807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2931.211760] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2936.771890] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2942.033550] Lustre: server umount lustre-MDT0000 complete [ 2949.955933] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 00:32:33 (1788755553) [ 2951.541296] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2953.191588] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 00:32:36 (1788755556) [ 2967.467410] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2976.233095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2981.675632] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2990.368322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2990.487281] Lustre: 123858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2990.616209] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2990.624264] Lustre: Skipped 1 previous similar message [ 2990.753488] Lustre: lustre-MDT0001: new disk, initializing [ 2990.954594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2990.976236] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2995.665225] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3001.068197] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3008.727859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3009.235993] Lustre: lustre-OST0000: new disk, initializing [ 3009.239761] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3009.248615] Lustre: 125435:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3010.699394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3010.721785] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3010.926620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3017.450067] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3028.050771] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3028.316326] Lustre: lustre-OST0001: new disk, initializing [ 3028.321976] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3028.333569] Lustre: 126306:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3030.259388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3030.276853] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3030.424295] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3035.259284] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3044.928083] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3048.776410] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3429.223710] Lustre: 132721:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3429.234613] Lustre: 132721:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3429.240583] Lustre: 132721:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3429.246789] Lustre: 132721:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3429.254683] Lustre: 132721:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3429.260583] Lustre: 132721:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3715.331869] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:45:18 (1788756318) [ 3731.941507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3731.951038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3731.955135] Lustre: Skipped 6 previous similar messages [ 3731.987449] Lustre: Skipped 11 previous similar messages [ 3735.842271] Lustre: server umount lustre-MDT0000 complete [ 3737.057369] LustreError: 127107:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3737.089043] LustreError: 127107:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 13 previous similar messages [ 3739.419173] LustreError: 132734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756344 with bad export cookie 7071538328107194306 [ 3739.425921] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3739.446737] LustreError: 132734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3742.178339] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3742.182448] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3742.198652] Lustre: Skipped 3 previous similar messages [ 3742.219796] Lustre: Skipped 1 previous similar message [ 3745.977993] Lustre: server umount lustre-MDT0001 complete [ 3755.575663] Lustre: server umount lustre-OST0000 complete [ 3772.895399] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3773.124774] Lustre: server umount lustre-OST0001 complete [ 3780.086360] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 3788.980288] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 3824.134471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3824.332846] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3824.357077] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3824.456383] Lustre: lustre-MDT0000: new disk, initializing [ 3824.536763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3824.542447] Lustre: Skipped 5 previous similar messages [ 3824.554497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3828.651251] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3840.011462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3840.209243] Lustre: 135781:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3840.303359] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3840.307892] Lustre: Skipped 1 previous similar message [ 3840.414502] Lustre: lustre-MDT0001: new disk, initializing [ 3840.535932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3840.552352] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3846.801450] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3857.427513] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3863.510720] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3870.868592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3871.353815] Lustre: lustre-OST0000: new disk, initializing [ 3871.364328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3871.372095] Lustre: 137751:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3871.501466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3871.515608] Lustre: Skipped 1 previous similar message [ 3872.655581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3872.672449] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3872.779109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3878.505955] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3890.105461] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3900.687880] Lustre: lustre-MDD0000: changelog on [ 3903.505669] Lustre: lustre-MDD0001: changelog on [ 5664.152881] Lustre: lustre-MDD0001: changelog off [ 5665.664838] Lustre: 140072:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5666.367212] Lustre: lustre-MDD0000: changelog off [ 5674.978820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5674.979396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5674.984259] LustreError: Skipped 1 previous similar message [ 5674.998426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5675.001724] Lustre: Skipped 1 previous similar message [ 5675.015330] Lustre: Skipped 3 previous similar messages [ 5677.245079] Lustre: server umount lustre-OST0000 complete [ 5683.683942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5683.693622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5683.704202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5685.226933] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5686.717672] Lustre: server umount lustre-MDT0000 complete [ 5689.898380] LustreError: 135776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788758294 with bad export cookie 7071538328108018549 [ 5689.903782] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5689.920902] LustreError: 135776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5690.321561] Lustre: server umount lustre-MDT0001 complete [ 5700.320610] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 5702.746709] Key type lgssc unregistered [ 5702.989164] LNet: 146868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5702.996709] LNetError: 146868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5703.032770] LNet: Removed LNI 192.168.203.153@tcp [ 5703.893163] Key type .llcrypt unregistered [ 5703.897423] Key type ._llcrypt unregistered [ 5724.751994] Key type ._llcrypt registered [ 5724.757426] Key type .llcrypt registered [ 5724.892872] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 5740.706966] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 5741.944753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5741.993037] alg: No test for adler32 (adler32-zlib) [ 5743.076891] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 5743.312782] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 5744.992162] Key type lgssc registered [ 5745.958729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5779.716659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5779.762872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5780.956625] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5781.013902] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5781.073298] Lustre: lustre-MDT0000: new disk, initializing [ 5781.119848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5781.133086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5785.488523] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5795.424020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5795.576434] Lustre: 150066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5795.695943] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5795.701114] Lustre: Skipped 1 previous similar message [ 5795.800945] Lustre: lustre-MDT0001: new disk, initializing [ 5795.956352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5795.999453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5796.007703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5799.568879] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5807.857768] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5812.847723] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5818.856051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5819.131356] Lustre: lustre-OST0000: new disk, initializing [ 5819.136446] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5819.145609] Lustre: 152033:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5819.246845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5825.782479] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5826.662663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5826.694521] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5826.782139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5834.608133] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5840.682922] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_module obdecho/obdecho [ 5842.241846] LustreError: 153035:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5842.248553] LustreError: 153035:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5843.774189] LustreError: 153133:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5857.248849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5857.262387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5857.288824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5857.767368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5861.027605] Lustre: server umount lustre-MDT0000 complete [ 5864.522708] LustreError: 150060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788758469 with bad export cookie 3082796061785121188 [ 5864.535293] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5864.553035] LustreError: 150060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5864.991380] Lustre: server umount lustre-MDT0001 complete [ 5878.909749] Lustre: server umount lustre-OST0000 complete [ 5900.671718] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 5912.634161] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 5951.389066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5951.727258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5951.801429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5951.932242] Lustre: lustre-MDT0000: new disk, initializing [ 5952.038748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5952.065918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5957.332826] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5967.928696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5968.075996] Lustre: 156852:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5968.126130] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5968.135883] Lustre: Skipped 1 previous similar message [ 5968.215956] Lustre: lustre-MDT0001: new disk, initializing [ 5968.313759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5968.359455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5968.379485] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5972.844833] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5983.309694] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5989.014779] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5995.825345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5996.010502] Lustre: lustre-OST0000: new disk, initializing [ 5996.014131] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5996.019258] Lustre: 158821:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5996.096023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6002.791873] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6004.855693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6004.875522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6005.005127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6012.075485] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6017.133146] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6017.311560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6021.684797] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6021.940623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6025.704971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6025.706431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6025.728559] Lustre: Skipped 1 previous similar message [ 6025.738990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6025.745618] LustreError: Skipped 1 previous similar message [ 6025.751526] Lustre: Skipped 2 previous similar messages [ 6030.820882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6030.837239] Lustre: Skipped 1 previous similar message [ 6035.964349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6035.979298] Lustre: Skipped 2 previous similar messages [ 6037.983413] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6038.320239] Lustre: server umount lustre-OST0000 complete [ 6045.152597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6045.167676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6045.185448] Lustre: Skipped 1 previous similar message [ 6045.201768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6048.866790] Lustre: server umount lustre-MDT0000 complete [ 6051.296415] LustreError: 157742:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6051.331632] LustreError: 157742:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [ 6052.596778] LustreError: 158831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788758657 with bad export cookie 3082796061785122147 [ 6052.601854] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6052.607592] LustreError: 158831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6052.983254] Lustre: server umount lustre-MDT0001 complete [ 6056.573509] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 01:24:19 (1788758659) [ 6058.257831] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6060.032820] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 01:24:23 (1788758663) [ 6068.057251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6068.586271] LustreError: 161235:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6068.709746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6073.215180] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6073.826123] LustreError: 161236:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6078.974336] LustreError: 161235:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6083.010612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6088.694867] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6096.782580] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6101.686742] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6107.651876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6108.057962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6108.067508] Lustre: Skipped 1 previous similar message [ 6114.347491] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6121.170654] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6127.062346] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.53@tcp (stopping) [ 6127.073948] Lustre: Skipped 2 previous similar messages [ 6129.634337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6129.636588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6129.650677] Lustre: Skipped 3 previous similar messages [ 6132.498137] Lustre: server umount lustre-MDT0000 complete [ 6134.760816] LustreError: 161235:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6134.797187] LustreError: 161235:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 6135.601208] LustreError: 161217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788758740 with bad export cookie 3082796061785123029 [ 6135.602954] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6135.614605] LustreError: 161217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6139.879059] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6141.984836] Lustre: server umount lustre-MDT0001 complete [ 6147.699170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6159.671629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6159.880511] LustreError: 165073:0:(llog.c:1655:llog_backup()) MGC192.168.203.153@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6159.893095] Lustre: 165073:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.153@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6165.476633] LustreError: 165078:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6165.508655] LustreError: 165078:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 6181.859372] LustreError: 165080:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6181.889081] LustreError: 165080:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 11 previous similar messages [ 6217.699693] LustreError: 163066:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6217.732642] LustreError: 163066:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 27 previous similar messages [ 6221.280815] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6221.298997] Lustre: 165073:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.153@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6262.285271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6267.162503] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6267.381275] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6276.030536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6282.676909] LustreError: 165078:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6282.703905] LustreError: 165078:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 41 previous similar messages [ 6282.719581] LustreError: 148289:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a3c920d4 x1875649072594304/t0(0) o250->MGC192.168.203.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6287.097634] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6288.363186] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6294.708920] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6296.562601] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6302.279549] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6303.867683] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6324.213229] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6324.219307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6324.231245] Lustre: Skipped 1 previous similar message [ 6324.244078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6324.262192] Lustre: Skipped 9 previous similar messages [ 6326.792420] Lustre: server umount lustre-MDT0000 complete [ 6330.499414] LustreError: 161215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788758935 with bad export cookie 3082796061785129483 [ 6330.506456] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6330.511203] LustreError: 161215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6331.087533] Lustre: server umount lustre-MDT0001 complete [ 6345.383772] Lustre: server umount lustre-OST0000 complete [ 6365.598954] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 6376.568664] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6412.641111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6412.912719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6412.943119] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6413.030851] Lustre: lustre-MDT0000: new disk, initializing [ 6413.137905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6413.146459] Lustre: Skipped 1 previous similar message [ 6413.165345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6417.976397] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6428.904863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6429.027702] Lustre: 170179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6429.304935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6434.256509] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6443.294169] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6448.484874] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6456.109899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6456.369137] Lustre: lustre-OST0000: new disk, initializing [ 6456.371589] Lustre: Skipped 1 previous similar message [ 6456.374762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6456.378161] Lustre: Skipped 2 previous similar messages [ 6456.382066] Lustre: 172147:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6462.872499] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6466.153785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6466.169429] Lustre: Skipped 1 previous similar message [ 6466.177595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6466.352076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6473.406916] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6479.121593] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6479.426104] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6484.414844] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6484.660919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6487.016361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6487.027635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6487.044932] Lustre: Skipped 1 previous similar message [ 6487.063544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6492.976329] Lustre: server umount lustre-OST0000 complete [ 6503.366280] Lustre: server umount lustre-MDT0000 complete [ 6507.497691] LustreError: 170192:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6507.533188] LustreError: 170192:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 7 previous similar messages [ 6507.619138] LustreError: 170173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788759112 with bad export cookie 3082796061785146318 [ 6507.627317] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6507.637209] LustreError: 170173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6508.072591] Lustre: server umount lustre-MDT0001 complete [ 6519.227840] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 01:32:01 (1788759121) [ 6527.027496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6532.392391] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6542.172419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6548.209742] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6557.127666] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6563.252987] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6569.916355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6577.190273] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6584.373900] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6597.363567] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 01:33:20 (1788759200) [ 6603.234490] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6603.238440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6603.247067] LustreError: Skipped 1 previous similar message [ 6603.248929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6603.258119] Lustre: Skipped 3 previous similar messages [ 6603.269027] Lustre: Skipped 7 previous similar messages [ 6607.475609] Lustre: server umount lustre-OST0000 complete [ 6617.945847] Lustre: server umount lustre-MDT0000 complete [ 6622.376587] LustreError: 174538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788759227 with bad export cookie 3082796061785147193 [ 6622.412444] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6622.840348] Lustre: server umount lustre-MDT0001 complete [ 6633.798590] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 6636.479642] Key type lgssc unregistered [ 6636.701543] LNet: 178286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6636.708388] LNetError: 178286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6637.734186] LNet: Removed LNI 192.168.203.153@tcp [ 6638.947102] Key type .llcrypt unregistered [ 6638.950572] Key type ._llcrypt unregistered [ 6659.562778] Key type ._llcrypt registered [ 6659.572077] Key type .llcrypt registered [ 6659.732753] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6660.732542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6660.769341] alg: No test for adler32 (adler32-zlib) [ 6661.839526] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 6662.042492] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 6663.679461] Key type lgssc registered [ 6665.285495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6673.111813] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6673.126749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6688.863544] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6693.983966] LustreError: 179530:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.153@tcp: failed processing log, type 4: rc = -110 [ 6699.103920] LustreError: 179530:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6699.128932] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6699.140570] LustreError: MGC192.168.203.153@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6699.157799] LustreError: 179530:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6699.191597] LustreError: 179530:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 6699.212127] LustreError: 179530:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 6699.290385] Lustre: server umount lustre-OST0000 complete [ 6699.292632] LustreError: 179530:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6707.758760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6709.182264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6713.681029] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6718.442699] LustreError: 180031:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6718.473989] LustreError: 180031:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 6723.301704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6723.554751] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6723.750740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6728.298575] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6735.204884] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6740.453620] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6746.770207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6747.014528] LustreError: 181845:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6747.027806] Lustre: 181845:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.153@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6747.156767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6753.239961] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6759.706405] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6764.514972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6764.518404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6764.521471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6764.549294] Lustre: Skipped 2 previous similar messages [ 6767.673908] LustreError: 182608:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (000000001403cca7) refcount nonzero (0) after lock cleanup; forcing cleanup. [ 6767.855413] Lustre: server umount lustre-MDT0000 complete [ 6769.635896] LustreError: 180031:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6769.650959] LustreError: 180031:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 6770.626663] LustreError: 180010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788759375 with bad export cookie 3667254872791466135 [ 6770.628892] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6770.635207] LustreError: 180010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6770.951461] Lustre: server umount lustre-MDT0001 complete [ 6784.768180] Lustre: server umount lustre-OST0000 complete [ 6792.331877] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 6801.600995] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6832.639505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6832.869681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6832.897504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6832.976953] Lustre: lustre-MDT0000: new disk, initializing [ 6833.058822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6833.080920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6836.888627] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6845.649072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6845.747536] Lustre: 185370:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6845.786725] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6845.792347] Lustre: Skipped 1 previous similar message [ 6845.866973] Lustre: lustre-MDT0001: new disk, initializing [ 6845.952596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6845.978443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6845.997642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6849.735465] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6857.546780] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6861.752924] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6867.071894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6867.289795] Lustre: lustre-OST0000: new disk, initializing [ 6867.293216] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6867.301797] Lustre: 187336:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6867.370786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6872.685649] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6875.758891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6875.766384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6875.895974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6881.805638] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6886.064464] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6886.370094] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6890.450894] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6890.621044] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6896.610321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6896.614511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6896.643860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6896.646704] Lustre: Skipped 2 previous similar messages [ 6898.534536] Lustre: server umount lustre-OST0000 complete [ 6902.241439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6902.248138] LustreError: Skipped 1 previous similar message [ 6902.254578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6902.266365] Lustre: Skipped 1 previous similar message [ 6902.273443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6906.849470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6906.859545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6908.350492] Lustre: server umount lustre-MDT0000 complete [ 6911.974411] LustreError: 186432:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6911.993541] LustreError: 186432:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 6912.243958] LustreError: 185360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788759516 with bad export cookie 3667254872791466856 [ 6912.251997] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6912.257808] LustreError: 185360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6912.723418] Lustre: server umount lustre-MDT0001 complete [ 6921.238703] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 01:38:44 (1788759524) [ 6936.984285] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 6939.470691] Key type lgssc unregistered [ 6939.737669] LNet: 190279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6939.755288] LNetError: 190279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6939.775609] LNet: Removed LNI 192.168.203.153@tcp [ 6940.623588] Key type .llcrypt unregistered [ 6940.626042] Key type ._llcrypt unregistered [ 6956.635520] Key type ._llcrypt registered [ 6956.642083] Key type .llcrypt registered [ 6956.782232] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6957.908501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6957.929616] alg: No test for adler32 (adler32-zlib) [ 6959.060994] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 6959.267391] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 6960.975205] Key type lgssc registered [ 6962.223034] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6969.690528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6969.706084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6971.450127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6977.096177] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6981.601492] LustreError: 191494:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6981.620457] LustreError: 191494:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 6986.736038] LustreError: 191495:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6986.855717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6987.212846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6991.728454] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6999.249392] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7005.197668] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7012.549845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7012.804648] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7012.813162] LustreError: 193360:0:(llog.c:1696:llog_backup()) MGC192.168.203.153@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7012.823807] Lustre: 193360:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.153@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7012.983073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7018.698520] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7026.527214] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7031.526264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7031.528513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7031.560281] Lustre: Skipped 2 previous similar messages [ 7031.572922] Lustre: Skipped 2 previous similar messages [ 7034.825617] Lustre: server umount lustre-MDT0000 complete [ 7036.898105] LustreError: 191495:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7036.911708] LustreError: 191495:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 7038.290731] LustreError: 191476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788759642 with bad export cookie 11011202179350492458 [ 7038.298631] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7038.315404] LustreError: 191476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7039.776954] LustreError: 192224:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000032269477 x1875650347559936/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7040.044517] Lustre: server umount lustre-MDT0001 complete [ 7053.777204] Lustre: server umount lustre-OST0000 complete [ 7065.183467] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 7076.883847] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 7113.349105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7113.560588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7113.607511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7113.662683] Lustre: lustre-MDT0000: new disk, initializing [ 7113.737869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7113.757677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7118.206777] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7129.285985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7129.451062] Lustre: 196887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7129.490813] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7129.494606] Lustre: Skipped 1 previous similar message [ 7129.497504] LustreError: 197669:0:(llog.c:1696:llog_backup()) MGC192.168.203.153@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7129.507248] LustreError: 197669:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7129.512851] Lustre: 197669:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.153@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7129.544963] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7129.554037] Lustre: Skipped 1 previous similar message [ 7129.615441] Lustre: lustre-MDT0001: new disk, initializing [ 7129.727655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7129.766983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7129.784491] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7134.112202] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7142.948506] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7148.099320] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7153.916679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7154.256682] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7154.260534] Lustre: Skipped 1 previous similar message [ 7154.265805] LustreError: 198831:0:(llog.c:1696:llog_backup()) MGC192.168.203.153@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7154.282454] LustreError: 198831:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7154.287841] Lustre: 198831:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.153@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7154.300691] Lustre: 198831:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7154.325179] Lustre: lustre-OST0000: new disk, initializing [ 7154.330391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7154.338183] Lustre: 198856:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7154.429524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7159.705297] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7159.852177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7159.872580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7160.012931] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7168.716165] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7173.400792] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7173.654742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7178.437698] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7178.673319] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7180.768320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7180.782161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7180.799754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7180.811384] Lustre: Skipped 1 previous similar message [ 7185.895197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7185.900386] Lustre: Skipped 1 previous similar message [ 7186.856185] Lustre: server umount lustre-OST0000 complete [ 7191.023825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7191.035453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7191.046418] Lustre: Skipped 2 previous similar messages [ 7191.051250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7191.084339] LustreError: Skipped 1 previous similar message [ 7196.134724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7196.144022] Lustre: Skipped 2 previous similar messages [ 7196.668656] Lustre: server umount lustre-MDT0000 complete [ 7199.975588] LustreError: 198854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788759804 with bad export cookie 11011202179350493179 [ 7199.977794] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7199.991095] LustreError: 198854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7200.544838] Lustre: server umount lustre-MDT0001 complete [ 7208.883847] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 01:43:32 (1788759812) [ 7224.964951] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 7233.511482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7233.841026] LustreError: 201981:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7233.879995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7238.196680] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7239.142670] LustreError: 201982:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7246.799410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7251.730906] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7254.257322] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7260.417966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7267.006579] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7274.349455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7274.522907] Lustre: lustre-OST0001: new disk, initializing [ 7274.527340] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7274.534414] Lustre: 204285:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7274.608810] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7274.616200] Lustre: Skipped 2 previous similar messages [ 7279.697504] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7280.743130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7280.764407] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7280.851170] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7288.956283] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7294.631189] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7296.467355] Lustre: DEBUG MARKER: ADD OST3 [ 7301.364629] loop: module loaded [ 7301.472072] loop0: detected capacity change from 0 to 204800000 [ 7301.717334] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7301.728675] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7301.752156] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7308.008741] loop0: detected capacity change from 0 to 204800000 [ 7308.688638] Lustre: DEBUG MARKER: STOP OST3 [ 7313.742166] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7314.043467] Lustre: lustre-OST0003: new disk, initializing [ 7314.048249] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7314.060176] Lustre: 206312:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7315.810623] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7315.928295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7315.939204] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7315.988863] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7317.291672] Lustre: Failing over lustre-OST0003 [ 7317.483343] Lustre: server umount lustre-OST0003 complete [ 7317.857071] LustreError: 206488:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7320.547731] LustreError: 203464:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7320.581596] LustreError: 203464:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 7320.873388] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7321.085685] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7321.099783] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7327.791936] Lustre: DEBUG MARKER: START OST3 again [ 7333.686166] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7334.010491] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7335.144424] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7335.202119] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7335.208148] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7340.665649] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7361.496609] Lustre: Failing over lustre-OST0003 [ 7361.596265] Lustre: server umount lustre-OST0003 complete [ 7364.586847] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7364.603119] LustreError: 203467:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7364.612461] Lustre: Skipped 2 previous similar messages [ 7364.648344] LustreError: 203467:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 9 previous similar messages [ 7379.940600] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7379.950733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7379.957537] Lustre: Skipped 2 previous similar messages [ 7379.965541] Lustre: Skipped 3 previous similar messages [ 7385.509142] Lustre: server umount lustre-MDT0000 complete [ 7389.510985] LustreError: 201963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788759994 with bad export cookie 11011202179350494061 [ 7389.523781] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7389.526631] LustreError: 201963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7395.974554] Lustre: server umount lustre-MDT0001 complete [ 7400.263637] Lustre: server umount lustre-OST0000 complete [ 7404.365567] Lustre: server umount lustre-OST0001 complete [ 7410.486432] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 7417.932880] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 7448.572600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7448.765907] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7448.784762] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7448.842517] Lustre: lustre-MDT0000: new disk, initializing [ 7448.935703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7448.943611] Lustre: Skipped 2 previous similar messages [ 7448.959614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7452.623779] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7461.093719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7461.181581] Lustre: 211747:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7461.422056] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7465.333610] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7473.223263] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7477.818490] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7483.919707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7484.219688] Lustre: lustre-OST0000: new disk, initializing [ 7484.223798] Lustre: Skipped 1 previous similar message [ 7484.227520] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7484.232092] Lustre: Skipped 2 previous similar messages [ 7484.236375] Lustre: 213712:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7486.204287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7486.221369] Lustre: Skipped 1 previous similar message [ 7486.226090] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7486.290643] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7489.863399] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7498.251182] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7502.790892] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7503.036572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7507.544501] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7507.730593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7512.034622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7512.042239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7512.042570] LustreError: Skipped 1 previous similar message [ 7512.052608] Lustre: Skipped 3 previous similar messages [ 7512.056729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7512.056736] Lustre: Skipped 8 previous similar messages [ 7515.858454] Lustre: server umount lustre-OST0000 complete [ 7525.492542] Lustre: server umount lustre-MDT0000 complete [ 7527.394991] LustreError: 211757:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7527.431963] LustreError: 211757:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 11 previous similar messages [ 7529.235143] LustreError: 213711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788760133 with bad export cookie 11011202179350496357 [ 7529.238039] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7529.245432] LustreError: 213711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7529.557788] Lustre: server umount lustre-MDT0001 complete [ 7533.590738] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 01:48:56 (1788760136) [ 7550.480438] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 7560.089337] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 7589.543526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7589.765428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7589.805259] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7589.860651] Lustre: lustre-MDT0000: new disk, initializing [ 7589.910566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7589.916375] Lustre: Skipped 2 previous similar messages [ 7589.936640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7593.657975] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7603.610600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7603.701712] Lustre: 218319:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7603.996310] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7608.165542] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7616.399143] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7620.995969] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7626.275909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7626.557292] Lustre: 220288:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7628.301394] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7632.050987] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7640.442863] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7644.729340] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7644.977534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7648.781664] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7648.949738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7653.864325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7653.866717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7653.878429] LustreError: Skipped 1 previous similar message [ 7653.907627] Lustre: Skipped 2 previous similar messages [ 7653.909679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7653.925521] Lustre: Skipped 4 previous similar messages [ 7656.856172] Lustre: server umount lustre-OST0000 complete [ 7666.501166] Lustre: server umount lustre-MDT0000 complete [ 7669.220786] LustreError: 218325:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7669.245831] LustreError: 218325:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 7670.125185] LustreError: 220286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788760274 with bad export cookie 11011202179350497239 [ 7670.126954] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7670.135196] LustreError: 220286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7670.547713] Lustre: server umount lustre-MDT0001 complete [ 7678.066727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7682.896957] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7691.492870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7695.790963] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7701.775146] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7706.466675] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7712.878419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7719.367917] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7725.940246] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7736.800187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7736.816776] LustreError: Skipped 1 previous similar message [ 7736.828380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7736.848190] Lustre: Skipped 3 previous similar messages [ 7736.859386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7736.874372] Lustre: Skipped 3 previous similar messages [ 7741.314198] Lustre: server umount lustre-OST0000 complete [ 7750.806703] Lustre: server umount lustre-MDT0000 complete [ 7753.884208] LustreError: 222487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788760358 with bad export cookie 11011202179350498114 [ 7753.887075] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7753.893168] LustreError: 222487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7754.232759] Lustre: server umount lustre-MDT0001 complete [ 7863.359507] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 7865.578276] Key type lgssc unregistered [ 7865.839926] LNet: 226167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7865.850340] LNetError: 226167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7865.869614] LNet: Removed LNI 192.168.203.153@tcp [ 7866.648182] Key type .llcrypt unregistered [ 7866.650969] Key type ._llcrypt unregistered [ 7887.327449] Key type ._llcrypt registered [ 7887.331114] Key type .llcrypt registered [ 7887.409880] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 7901.717456] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 7902.590355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7902.674715] alg: No test for adler32 (adler32-zlib) [ 7903.739250] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 7904.033619] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 7905.703162] Key type lgssc registered [ 7906.803482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7939.187836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7939.223367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7940.524176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7940.545207] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7940.628169] Lustre: lustre-MDT0000: new disk, initializing [ 7940.709205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7940.729953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7944.729980] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7955.840706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7955.984513] Lustre: 229511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7956.061135] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7956.067599] Lustre: Skipped 1 previous similar message [ 7956.178126] Lustre: lustre-MDT0001: new disk, initializing [ 7956.310106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7956.344579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7956.366137] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7961.184338] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7971.522610] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7977.033528] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7983.374565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7983.646852] Lustre: lustre-OST0000: new disk, initializing [ 7983.652547] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7983.666692] Lustre: 231481:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7983.755759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7990.005697] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 7993.441063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7993.464462] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7993.582774] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7999.166699] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8003.830358] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8004.042785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8008.414632] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8008.746175] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8014.311283] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8014.321292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8014.323128] LustreError: Skipped 1 previous similar message [ 8014.357738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8014.370618] Lustre: Skipped 1 previous similar message [ 8017.238103] Lustre: server umount lustre-OST0000 complete [ 8023.008301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8023.019111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8023.030682] Lustre: Skipped 1 previous similar message [ 8023.039552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8024.547472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8024.555566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8026.815521] Lustre: server umount lustre-MDT0000 complete [ 8029.928193] LustreError: 229504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788760634 with bad export cookie 11194969714428009265 [ 8029.935034] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8029.941699] LustreError: 229504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8030.232372] Lustre: server umount lustre-MDT0001 complete [ 8047.774324] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 8057.454309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8057.810340] LustreError: 234415:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8057.829255] LustreError: 234415:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 8057.900126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8062.166331] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8062.951830] LustreError: 234416:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8068.069075] LustreError: 234415:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8070.269336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8070.785693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8075.121816] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8077.925557] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8084.367457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8084.585443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8089.958950] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8097.956816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8098.133284] Lustre: lustre-OST0001: new disk, initializing [ 8098.137362] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8098.145716] Lustre: 236721:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8103.867655] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8107.706028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8107.725568] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8107.899756] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8116.379125] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8127.692054] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8140.934877] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 01:59:03 (1788760743) [ 8143.845375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8143.862833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8143.869700] Lustre: Skipped 2 previous similar messages [ 8143.875569] Lustre: Skipped 2 previous similar messages [ 8148.960948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8148.976651] Lustre: Skipped 3 previous similar messages [ 8157.151131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8157.365389] Lustre: server umount lustre-MDT0000 complete [ 8159.208027] LustreError: 237503:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8159.229289] LustreError: 237503:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 8160.698256] LustreError: 234397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788760765 with bad export cookie 11194969714428010147 [ 8160.713665] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8161.239702] Lustre: server umount lustre-MDT0001 complete [ 8175.211210] Lustre: server umount lustre-OST0000 complete [ 8188.915822] Lustre: server umount lustre-OST0001 complete [ 8195.089366] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 8204.390915] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 8239.131224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8239.423386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8239.455922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8239.553980] Lustre: lustre-MDT0000: new disk, initializing [ 8239.651668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8239.656862] Lustre: Skipped 1 previous similar message [ 8239.671342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8244.457844] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8255.242533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8255.369575] Lustre: 240671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8255.426411] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8255.430472] Lustre: Skipped 1 previous similar message [ 8255.524216] Lustre: lustre-MDT0001: new disk, initializing [ 8255.754289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8255.773271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8260.296730] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8271.264308] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8276.547474] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8283.256227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8283.552436] Lustre: lustre-OST0000: new disk, initializing [ 8283.555994] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8283.562898] Lustre: 242650:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8283.645985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8283.649536] Lustre: Skipped 1 previous similar message [ 8284.807432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8284.819209] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8284.925076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8289.388975] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8298.393909] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8302.994635] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8303.216899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8307.832580] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8308.058511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8310.765375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8310.770740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8310.774570] LustreError: Skipped 1 previous similar message [ 8310.776501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8310.793225] Lustre: Skipped 2 previous similar messages [ 8310.808724] Lustre: Skipped 5 previous similar messages [ 8324.063232] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8324.244402] Lustre: server umount lustre-OST0000 complete [ 8331.241567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8331.262485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8331.267629] Lustre: Skipped 1 previous similar message [ 8331.308668] Lustre: Skipped 5 previous similar messages [ 8334.393913] Lustre: server umount lustre-MDT0000 complete [ 8336.357363] LustreError: 240679:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8336.391312] LustreError: 240679:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 8338.224807] LustreError: 241466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788760942 with bad export cookie 11194969714428012275 [ 8338.228600] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8338.233559] LustreError: 241466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8338.604838] Lustre: server umount lustre-MDT0001 complete [ 8346.498348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8347.005427] LustreError: 244869:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8351.578545] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8359.802502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8360.279899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8360.288346] Lustre: Skipped 1 previous similar message [ 8365.034020] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8374.268438] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8382.217266] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8390.237515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8397.055362] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8403.768868] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8553.954067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8553.958687] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8553.980635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8557.587146] Lustre: server umount lustre-OST0000 complete [ 8568.385003] Lustre: server umount lustre-MDT0000 complete [ 8569.312484] LustreError: 246110:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8569.333155] LustreError: 246110:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 8571.725172] LustreError: 244850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788761176 with bad export cookie 11194969714428013164 [ 8571.727438] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8571.756759] LustreError: 244850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8571.994943] Lustre: server umount lustre-MDT0001 complete [ 8670.554127] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 8672.858479] Key type lgssc unregistered [ 8673.083636] LNet: 248473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8673.089379] LNetError: 248473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8673.112809] LNet: Removed LNI 192.168.203.153@tcp [ 8673.913158] Key type .llcrypt unregistered [ 8673.916818] Key type ._llcrypt unregistered [ 8697.471929] Key type ._llcrypt registered [ 8697.475693] Key type .llcrypt registered [ 8697.576800] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 8713.773690] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 8714.999602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8715.085254] alg: No test for adler32 (adler32-zlib) [ 8716.122648] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 8716.366358] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 8718.041376] Key type lgssc registered [ 8719.100802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8763.234291] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 8772.630485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8772.689859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8773.973789] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8773.995224] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8774.070450] Lustre: lustre-MDT0000: new disk, initializing [ 8774.152753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8774.168564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8778.637957] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8789.069969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8789.239919] Lustre: 252538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8789.311656] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8789.315770] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8789.332846] Lustre: Skipped 1 previous similar message [ 8789.423311] Lustre: lustre-MDT0001: new disk, initializing [ 8789.571138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8789.673928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8789.712379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8794.086179] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8798.467725] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8804.560274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8804.827816] Lustre: lustre-OST0000: new disk, initializing [ 8804.835157] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8804.842324] Lustre: 254170:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8804.941699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8809.542708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8809.562172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8809.666036] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8810.557403] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8822.078979] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8822.282279] Lustre: lustre-OST0001: new disk, initializing [ 8822.290977] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8822.304232] Lustre: 255041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8822.421294] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8829.052232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8829.064529] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8829.153352] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8829.423575] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8840.299973] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8847.593697] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8860.130106] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8860.137298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8860.160368] Lustre: Skipped 1 previous similar message [ 8860.185699] Lustre: Skipped 3 previous similar messages [ 8865.260680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8865.282510] Lustre: Skipped 3 previous similar messages [ 8866.305110] Lustre: server umount lustre-MDT0000 complete [ 8870.492364] LustreError: 252531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788761475 with bad export cookie 13234397146531049679 [ 8870.499608] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8870.509148] LustreError: 252531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8870.845800] Lustre: server umount lustre-MDT0001 complete [ 8884.827231] Lustre: server umount lustre-OST0000 complete [ 8897.563951] Lustre: server umount lustre-OST0001 complete [ 8912.548362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8923.824752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8933.576196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8944.385767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8964.220619] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 8974.405676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8974.426648] Lustre: lustre-MDT0000: reset Object Index mappings [ 8974.836721] LustreError: 260054:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8974.849037] LustreError: 260054:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [ 8974.893942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8978.746912] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8979.942424] LustreError: 260055:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8985.057208] LustreError: 260054:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8986.449094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8986.481946] Lustre: lustre-MDT0001: reset Object Index mappings [ 8986.921750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8991.080206] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 8993.329637] Lustre: 261197:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8999.302497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8999.405696] LustreError: 261549:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9005.025890] LustreError: 261551:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9005.043441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9005.054488] LustreError: 261551:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 9006.284556] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9013.295256] LustreError: 261551:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9013.317473] LustreError: 261551:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 9014.584789] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9014.874543] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9014.881752] Lustre: Skipped 1 previous similar message [ 9019.905944] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9020.973948] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9029.070711] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9032.389106] Lustre: 263067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9038.305564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9038.310720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9038.319610] Lustre: Skipped 2 previous similar messages [ 9038.333781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9040.366623] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9040.388879] Lustre: Skipped 2 previous similar messages [ 9042.896326] Lustre: server umount lustre-MDT0000 complete [ 9045.476242] LustreError: 260050:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9045.519408] LustreError: 260050:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 9046.671781] LustreError: 260036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788761651 with bad export cookie 13234397146531056476 [ 9046.682319] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9046.694423] LustreError: 260036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9047.124143] Lustre: server umount lustre-MDT0001 complete [ 9060.489108] Lustre: server umount lustre-OST0000 complete [ 9074.107614] Lustre: server umount lustre-OST0001 complete [ 9089.352973] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 02:14:52 (1788761692) [ 9106.507273] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 9116.249798] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 9145.685459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9146.041355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9146.076103] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9146.141251] Lustre: lustre-MDT0000: new disk, initializing [ 9146.229191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9146.260877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9150.346804] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9159.609420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9159.760889] Lustre: 267319:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9159.848237] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9159.852722] Lustre: Skipped 1 previous similar message [ 9159.932483] Lustre: lustre-MDT0001: new disk, initializing [ 9160.066520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9160.079532] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9163.703981] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9172.085295] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9177.195598] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9183.599346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9183.846620] Lustre: lustre-OST0000: new disk, initializing [ 9183.850933] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9183.857693] Lustre: 269289:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9185.152940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9185.172867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9185.268767] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9189.425635] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9198.445512] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9203.353653] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9203.653460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9209.063137] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9209.258488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9215.975487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9216.004112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9216.012308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9216.017476] Lustre: Skipped 4 previous similar messages [ 9217.365270] Lustre: server umount lustre-OST0000 complete [ 9221.088442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9221.096384] Lustre: Skipped 1 previous similar message [ 9226.209649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9226.225348] Lustre: Skipped 4 previous similar messages [ 9227.113528] Lustre: server umount lustre-MDT0000 complete [ 9230.950901] LustreError: 267314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788761835 with bad export cookie 13234397146531058569 [ 9230.952868] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9230.978558] LustreError: 267314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9231.698828] Lustre: server umount lustre-MDT0001 complete [ 9252.319841] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 9262.055610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9262.416811] LustreError: 272225:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9262.458823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9262.474163] Lustre: Skipped 2 previous similar messages [ 9266.238487] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9274.147768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9278.605238] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9281.178853] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9286.967646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9293.181449] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9301.386820] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9301.552843] Lustre: lustre-OST0001: new disk, initializing [ 9301.566319] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9301.574276] Lustre: 274532:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9302.975796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9302.988296] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9303.119670] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9307.727178] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9316.669853] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9319.683828] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9323.838903] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9323.946410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9333.731482] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9333.735618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9333.754122] Lustre: Skipped 3 previous similar messages [ 9333.757513] Lustre: Skipped 1 previous similar message [ 9338.664038] Lustre: server umount lustre-MDT0000 complete [ 9338.854301] LustreError: 272222:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9338.869679] LustreError: 272222:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [ 9342.252319] LustreError: 272205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788761946 with bad export cookie 13234397146531059451 [ 9342.271789] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9342.628401] Lustre: server umount lustre-MDT0001 complete [ 9357.380079] Lustre: server umount lustre-OST0000 complete [ 9360.159274] Lustre: 250048:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788761948/real 1788761948] req@ffff95aaebc62d80 x1875652191165184/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788761964 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9360.197102] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9360.217961] Lustre: Skipped 1 previous similar message [ 9361.102810] Lustre: server umount lustre-OST0001 complete [ 9375.175158] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 9383.692351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9387.502627] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9396.201151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9396.779655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9396.789128] Lustre: Skipped 4 previous similar messages [ 9402.042261] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9405.780412] Lustre: 279020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9413.450535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9420.538444] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9428.488641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9429.271826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9434.640351] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9435.862556] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9443.331961] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9446.938533] Lustre: 280892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9451.936914] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9451.974693] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9464.465967] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 02:21:07 (1788762067) [ 9480.681189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9480.695116] Lustre: Skipped 2 previous similar messages [ 9480.704674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9480.720540] Lustre: Skipped 5 previous similar messages [ 9486.511259] Lustre: server umount lustre-MDT0000 complete [ 9489.851971] LustreError: 280895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788762094 with bad export cookie 13234397146531066059 [ 9489.858115] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9489.860436] LustreError: 280895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9490.917113] LustreError: 277874:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9490.940466] LustreError: 277874:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 8 previous similar messages [ 9504.224540] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9504.528024] Lustre: server umount lustre-MDT0001 complete [ 9514.138471] Lustre: server umount lustre-OST0000 complete [ 9517.515725] Lustre: server umount lustre-OST0001 complete [ 9523.687020] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 9532.581290] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 9577.911662] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 9590.761422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9590.953155] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9590.973042] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9591.020513] Lustre: lustre-MDT0000: new disk, initializing [ 9591.113312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9595.510521] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9605.164696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9605.257608] Lustre: 285439:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9605.501204] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9609.066293] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9619.077913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9619.317433] Lustre: 287070:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9620.940710] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9625.036637] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9635.031285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9635.195129] Lustre: 287942:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9637.261817] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9641.198161] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9650.682715] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9654.379788] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9654.388799] Lustre: Skipped 1 previous similar message [ 9668.855830] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 02:24:32 (1788762272) [ 9672.163914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9672.173868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9672.186126] Lustre: Skipped 2 previous similar messages [ 9672.195516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9672.204901] Lustre: Skipped 10 previous similar messages [ 9677.410502] Lustre: server umount lustre-MDT0000 complete [ 9680.693442] LustreError: 285430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788762285 with bad export cookie 13234397146531073766 [ 9680.704907] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9680.717497] LustreError: 285430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9681.026466] Lustre: server umount lustre-MDT0001 complete [ 9694.821422] Lustre: server umount lustre-OST0000 complete [ 9707.834965] Lustre: server umount lustre-OST0001 complete [ 9717.254052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9717.945827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9717.949640] Lustre: Skipped 6 previous similar messages [ 9722.397571] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9729.793939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9734.000268] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9740.252972] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9745.615488] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9752.151409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9754.682157] LustreError: 292015:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9754.702244] LustreError: 292015:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 11 previous similar messages [ 9759.497375] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9766.736250] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9783.892278] loop1: detected capacity change from 0 to 204800000 [ 9784.128954] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9784.147109] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9784.172184] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9789.142595] loop1: detected capacity change from 0 to 204800000 [ 9794.674397] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9794.717832] LustreError: 290177:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs10668: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9794.728828] LustreError: 290177:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs10668 [ 9794.734223] LustreError: 290177:0:(mgs_handler.c:517:mgs_target_reg()) fs10668-MDT0001: Error checking target index: rc = -13 [ 9794.741447] LustreError: fs10668-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9794.750291] LustreError: 293907:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 9794.757774] LustreError: 293907:0:(tgt_mount.c:2128:server_put_super()) no obd fs10668-MDT0001 [ 9794.766229] LustreError: 293907:0:(tgt_mount.c:116:server_deregister_mount()) fs10668-MDT0001 not registered [ 9794.813953] Lustre: server umount fs10668-MDT0001 complete [ 9794.818429] LustreError: 293907:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9803.113331] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9803.189362] Lustre: Setting parameter fs10668-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs10668-MDT0001 [ 9803.245332] Lustre: srv-fs10668-MDT0001: No data found on store. Initialize space. [ 9803.249203] Lustre: Skipped 4 previous similar messages [ 9803.330566] Lustre: fs10668-MDT0001: new disk, initializing [ 9803.333626] Lustre: Skipped 3 previous similar messages [ 9803.430468] Lustre: srv-fs10668-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9803.442235] Lustre: 294373:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10668-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9803.461908] LustreError: 294373:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10668-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9808.458295] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9815.542826] Lustre: server umount fs10668-MDT0001 complete [ 9831.150211] loop1: detected capacity change from 0 to 204800000 [ 9837.822081] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9838.365300] Lustre: 295942:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10668-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9838.378162] LustreError: 295942:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10668-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9839.627480] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 9844.120800] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9844.707111] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 9848.933934] Lustre: server umount fs10668-MDT0001 complete [ 9859.555980] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.53@tcp (stopping) [ 9859.566946] Lustre: Skipped 4 previous similar messages [ 9860.066138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9860.079069] Lustre: Skipped 3 previous similar messages [ 9865.528754] Lustre: server umount lustre-MDT0000 complete [ 9876.049146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9876.211507] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9881.646546] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9882.110239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9891.300441] loop1: detected capacity change from 0 to 204800000 [ 9897.335304] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9897.877968] Lustre: 298389:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10668-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9897.888585] LustreError: 298389:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10668-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9901.971286] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 9907.345162] Lustre: server umount fs10668-MDT0001 complete [ 9932.769269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9935.704217] Lustre: server umount lustre-OST0000 complete [ 9945.785974] Lustre: server umount lustre-MDT0000 complete [ 9949.237950] LustreError: 290168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788762553 with bad export cookie 13234397146531076447 [ 9949.244281] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9949.259243] LustreError: 290168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9949.605794] Lustre: server umount lustre-MDT0001 complete [ 9960.517293] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 9963.351160] Key type lgssc unregistered [ 9963.715776] LNet: 300733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9963.720330] LNetError: 300733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9963.748292] LNet: Removed LNI 192.168.203.153@tcp [ 9964.646845] Key type .llcrypt unregistered [ 9964.648867] Key type ._llcrypt unregistered [ 9982.727894] Key type ._llcrypt registered [ 9982.730635] Key type .llcrypt registered [ 9982.815405] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 9983.955357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9984.128897] alg: No test for adler32 (adler32-zlib) [ 9985.183569] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [ 9985.403273] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 9987.079218] Key type lgssc registered [ 9988.296140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9996.113364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9996.137949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9997.604984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10001.531796] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10008.033089] LustreError: 302003:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10008.055198] LustreError: 302003:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [10009.997956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10010.407058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10015.079306] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10022.896675] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10028.206397] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10034.865690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10035.181427] LustreError: 303830:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10035.301874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10040.315390] LustreError: 303831:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10042.055826] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10042.411155] LustreError: 304099:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10042.443964] LustreError: 304099:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [10047.458361] LustreError: 303830:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10047.477812] LustreError: 303830:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [10050.009160] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10057.154961] LustreError: 303830:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10057.209084] LustreError: 303830:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [10070.137773] loop1: detected capacity change from 0 to 204800000 [10070.408350] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10070.418255] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10070.455051] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10077.627295] LustreError: 303831:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10077.670849] LustreError: 303831:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 11 previous similar messages [10078.022555] loop1: detected capacity change from 0 to 204800000 [10085.006580] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10085.241883] Lustre: fs20743-OST0000: new disk, initializing [10085.248167] Lustre: srv-fs20743-OST0000: No data found on store. Initialize space. [10085.255905] Lustre: 305736:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10085.356159] Lustre: fs20743-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10091.875616] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10101.064984] Lustre: server umount fs20743-OST0000 complete [10113.479245] LustreError: 303830:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10113.509116] LustreError: 303830:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 20 previous similar messages [10115.452304] loop1: detected capacity change from 0 to 204800000 [10119.143180] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10121.817169] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10122.297236] Lustre: fs20743-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10124.268466] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10129.107645] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10138.603805] loop2: detected capacity change from 0 to 204800000 [10138.930468] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10138.947987] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10138.970555] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10143.511729] loop2: detected capacity change from 0 to 204800000 [10149.215977] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10149.354626] LustreError: fs20743-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10149.377387] LustreError: fs20743-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10149.386852] LustreError: 308775:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10149.394966] LustreError: 308775:0:(tgt_mount.c:2128:server_put_super()) no obd fs20743-OST0001 [10149.399519] LustreError: 308775:0:(tgt_mount.c:116:server_deregister_mount()) fs20743-OST0001 not registered [10149.438042] Lustre: server umount fs20743-OST0001 complete [10149.442306] LustreError: 308775:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10159.396793] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10159.580745] Lustre: fs20743-OST0001: new disk, initializing [10159.589841] Lustre: srv-fs20743-OST0001: No data found on store. Initialize space. [10159.595559] Lustre: 309326:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10159.687479] Lustre: fs20743-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10166.229100] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10174.765956] Lustre: server umount fs20743-OST0000 complete [10180.022432] LustreError: 303830:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10180.050706] LustreError: 303830:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 38 previous similar messages [10183.580616] Lustre: server umount fs20743-OST0001 complete [10195.625144] loop1: detected capacity change from 0 to 204800000 [10202.037473] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10202.393538] Lustre: fs20743-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10208.714766] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10215.906499] loop2: detected capacity change from 0 to 204800000 [10221.805471] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10228.137683] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10236.824720] Lustre: server umount fs20743-OST0000 complete [10245.907388] Lustre: server umount fs20743-OST0001 complete [10260.456336] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10260.462959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10260.477439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10262.500398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10262.509842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10262.523303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10267.617970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10267.634697] Lustre: Skipped 2 previous similar messages [10272.742568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10272.743657] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10272.748234] Lustre: Skipped 1 previous similar message [10273.247140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10273.468889] Lustre: server umount lustre-OST0000 complete [10277.856671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10277.866645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10277.882337] Lustre: Skipped 1 previous similar message [10277.902914] Lustre: Skipped 1 previous similar message [10283.937444] Lustre: server umount lustre-MDT0000 complete [10287.787489] LustreError: 301983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788762892 with bad export cookie 5610155269718237827 [10287.792931] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10287.816378] LustreError: 301983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10288.151800] Lustre: server umount lustre-MDT0001 complete [10301.800183] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [10305.352761] Key type lgssc unregistered [10305.854757] LNet: 315250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10305.867649] LNetError: 315250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10305.903154] LNet: Removed LNI 192.168.203.153@tcp [10307.473169] Key type .llcrypt unregistered [10307.475083] Key type ._llcrypt unregistered [10341.644262] Key type ._llcrypt registered [10341.649183] Key type .llcrypt registered [10341.760521] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [10356.195786] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [10357.051532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10357.118511] alg: No test for adler32 (adler32-zlib) [10358.162433] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [10358.401144] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [10360.047337] Key type lgssc registered [10361.223968] Lustre: Echo OBD driver; http://www.lustre.org/ [10397.231817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10397.270447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10398.479709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10398.505558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10398.588078] Lustre: lustre-MDT0000: new disk, initializing [10398.662292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10398.681581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10402.670846] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10413.522672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10413.675820] Lustre: 319107:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10413.765197] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10413.771651] Lustre: Skipped 1 previous similar message [10413.848067] Lustre: lustre-MDT0001: new disk, initializing [10413.980192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10414.048276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10414.067773] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10419.082689] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10428.470355] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10434.412652] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10440.913207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10441.118818] Lustre: lustre-OST0000: new disk, initializing [10441.122494] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10441.130439] Lustre: 321075:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10441.210668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10447.026775] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10449.940426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10449.953526] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10450.057692] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10455.560821] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10457.454729] Lustre: Mounted lustre-client - version 2.17.58_43_g6a42d2c [10458.505434] Lustre: Unmounted lustre-client [10467.816670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10467.840825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10467.852100] Lustre: Skipped 1 previous similar message [10467.865340] Lustre: Skipped 1 previous similar message [10472.389551] Lustre: server umount lustre-OST0000 complete [10478.051593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10478.066688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10478.078423] Lustre: Skipped 1 previous similar message [10483.172075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10483.181583] Lustre: Skipped 1 previous similar message [10488.290401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10488.300054] Lustre: Skipped 1 previous similar message [10490.335149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10490.564610] Lustre: server umount lustre-MDT0000 complete [10494.518996] LustreError: 319099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788763099 with bad export cookie 3823648205228561691 [10494.530713] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10494.533456] LustreError: 319099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10494.987723] Lustre: server umount lustre-MDT0001 complete [10507.237437] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [10509.734414] Key type lgssc unregistered [10514.081290] LNet: 323240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10514.102965] LNetError: 323240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10589.938772] LNet: Removed LNI 192.168.203.153@tcp [10590.901237] Key type .llcrypt unregistered [10590.903986] Key type ._llcrypt unregistered [10607.718868] Key type ._llcrypt registered [10607.723587] Key type .llcrypt registered [10607.862992] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [10623.137950] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [10624.792291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10624.832228] alg: No test for adler32 (adler32-zlib) [10625.854957] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [10626.122176] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [10627.828837] Key type lgssc registered [10628.922166] Lustre: Echo OBD driver; http://www.lustre.org/ [10658.172814] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 02:41:01 (1788763261) [10664.660735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10664.689780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10666.005195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10666.032322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10666.165499] Lustre: lustre-MDT0000: new disk, initializing [10666.277438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10666.311061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10670.644721] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10681.240869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10681.408502] Lustre: 326456:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10681.504995] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10681.509254] Lustre: Skipped 1 previous similar message [10681.600162] Lustre: lustre-MDT0001: new disk, initializing [10681.712257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10681.772135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10681.791340] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10686.330983] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10695.182363] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10700.798783] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10708.440224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10708.793604] Lustre: lustre-OST0000: new disk, initializing [10708.801794] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10708.812715] Lustre: 328424:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10708.943408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10714.164418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10714.180871] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10714.334602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10715.286676] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10724.600702] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10740.197248] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10740.206969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10740.211908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10740.216876] LustreError: Skipped 1 previous similar message [10740.247880] Lustre: Skipped 1 previous similar message [10745.475122] Lustre: server umount lustre-OST0000 complete [10750.432895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10750.455593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10750.460659] Lustre: Skipped 2 previous similar messages [10754.995141] Lustre: server umount lustre-MDT0000 complete [10755.563705] LustreError: 326468:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10755.599035] LustreError: 326468:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [10758.725340] LustreError: 326448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788763363 with bad export cookie 12802734811507428342 [10758.727213] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10758.735110] LustreError: 326448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10759.090884] Lustre: server umount lustre-MDT0001 complete [10771.036412] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [10773.778233] Key type lgssc unregistered [10774.014559] LNet: 330329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10774.022063] LNetError: 330329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10774.041211] LNet: Removed LNI 192.168.203.153@tcp [10774.912765] Key type .llcrypt unregistered [10774.916856] Key type ._llcrypt unregistered [10796.759212] Key type ._llcrypt registered [10796.767335] Key type .llcrypt registered [10796.907314] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [10798.488063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10798.530249] alg: No test for adler32 (adler32-zlib) [10799.583853] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [10799.885237] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [10801.641142] Key type lgssc registered [10802.798793] Lustre: Echo OBD driver; http://www.lustre.org/ [10810.686657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10810.738159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10812.314881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10817.417814] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10821.607652] LustreError: 331605:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10821.626551] LustreError: 331605:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [10826.572278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10826.720169] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10826.946086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10831.542429] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10839.222324] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10845.360242] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10852.121120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10852.405375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10858.234116] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10865.115431] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10880.227618] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10881.715200] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10887.142872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10887.149656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10887.158103] Lustre: Skipped 2 previous similar messages [10887.173129] Lustre: Skipped 2 previous similar messages [10888.649697] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.53@tcp (stopping) [10891.351643] Lustre: server umount lustre-MDT0000 complete [10892.264238] LustreError: 331600:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10892.288587] LustreError: 331600:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [10893.750863] LustreError: 332620:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10894.560200] LustreError: 331585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788763499 with bad export cookie 8726679612835569566 [10894.564963] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10894.569772] LustreError: 331585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10897.377032] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10897.378050] LustreError: 331602:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10897.402513] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10901.051147] Lustre: server umount lustre-MDT0001 complete [10904.749910] Lustre: server umount lustre-OST0000 complete [10913.481378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10918.134488] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10920.903295] Lustre: 335784:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.153@tcp [10923.371634] Lustre: server umount MGS complete [10932.357300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10932.883714] LustreError: 336356:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10932.901723] LustreError: 336356:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [10932.969338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10937.490944] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10942.436028] LustreError: 336357:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10942.450748] LustreError: 336357:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [10946.457996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10946.764722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10951.691598] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10960.898092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10961.292781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10968.170029] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [10976.569664] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10978.192788] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [10984.373577] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10985.986524] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10992.100505] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10992.114822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10992.123507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10992.129851] Lustre: Skipped 1 previous similar message [10992.133723] Lustre: Skipped 2 previous similar messages [10992.983635] Lustre: server umount lustre-OST0000 complete [10998.244382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10998.250161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10998.261190] LustreError: Skipped 1 previous similar message [11003.364815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11003.370605] Lustre: Skipped 4 previous similar messages [11004.290176] Lustre: server umount lustre-MDT0000 complete [11007.464649] LustreError: 336356:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11007.494046] LustreError: 336356:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [11007.884513] LustreError: 337092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788763612 with bad export cookie 8726679612835570406 [11007.887786] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11007.893754] LustreError: 337092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11008.312843] Lustre: server umount lustre-MDT0001 complete [11019.286583] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [11022.114930] Key type lgssc unregistered [11022.412859] LNet: 339792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11022.427337] LNetError: 339792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11022.446736] LNet: Removed LNI 192.168.203.153@tcp [11023.422564] Key type .llcrypt unregistered [11023.427439] Key type ._llcrypt unregistered [11042.269988] Key type ._llcrypt registered [11042.273502] Key type .llcrypt registered [11042.402210] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [11060.180521] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [11061.408927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11061.539116] alg: No test for adler32 (adler32-zlib) [11062.691716] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [11062.997878] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [11064.807209] Key type lgssc registered [11066.132963] Lustre: Echo OBD driver; http://www.lustre.org/ [11102.357773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11102.401128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11103.609578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11103.659335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11103.756161] Lustre: lustre-MDT0000: new disk, initializing [11103.871611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11103.892682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11109.124596] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11119.902134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11120.052333] Lustre: 342953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11120.154768] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11120.161881] Lustre: Skipped 1 previous similar message [11120.249820] Lustre: lustre-MDT0001: new disk, initializing [11120.353445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11120.398300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11120.412681] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11126.004230] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11135.868902] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11141.374198] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11148.005661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11148.159657] Lustre: lustre-OST0000: new disk, initializing [11148.162735] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11148.168241] Lustre: 344921:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11148.273885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11154.060110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11154.083574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11154.222448] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11156.060251] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11167.885793] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11173.705709] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11173.989778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11179.105099] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11179.417227] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11185.130204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11185.139882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11185.168728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11185.180677] Lustre: Skipped 1 previous similar message [11190.246035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11195.379403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11195.382613] Lustre: Skipped 2 previous similar messages [11196.383160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11197.030135] Lustre: server umount lustre-OST0000 complete [11202.528555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11202.545467] LustreError: Skipped 1 previous similar message [11202.552243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11202.581550] Lustre: Skipped 1 previous similar message [11202.598116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11205.605309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11208.496936] Lustre: server umount lustre-MDT0000 complete [11213.573885] LustreError: 342945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788763818 with bad export cookie 7976234130659340492 [11213.577267] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11213.589252] LustreError: 342945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11214.321696] Lustre: server umount lustre-MDT0001 complete [11227.634335] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 02:50:29 (1788763829) [11246.062789] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [11256.738410] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [11297.480310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11297.656894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11297.676723] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11297.758885] Lustre: lustre-MDT0000: new disk, initializing [11297.822812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11297.856573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11301.958218] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11312.266400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11312.383454] Lustre: 349680:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11312.440177] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11312.447864] Lustre: Skipped 1 previous similar message [11312.557175] Lustre: lustre-MDT0001: new disk, initializing [11312.666770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11312.712286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11312.722721] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11317.425666] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11326.951677] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11332.696427] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11339.279534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11339.575300] Lustre: lustre-OST0000: new disk, initializing [11339.579414] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11339.586665] Lustre: 351650:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11339.708695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11345.520189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11345.544164] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11345.654131] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11346.658030] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [11356.248357] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11402.223904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11402.227894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11402.238520] LustreError: Skipped 1 previous similar message [11402.250278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11402.260314] Lustre: Skipped 1 previous similar message [11402.268768] Lustre: Skipped 2 previous similar messages [11404.219539] Lustre: server umount lustre-OST0000 complete [11409.896600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11409.905816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11412.448661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11412.456786] Lustre: Skipped 3 previous similar messages [11414.744935] Lustre: server umount lustre-MDT0000 complete [11417.571672] LustreError: 349693:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11417.617844] LustreError: 349693:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [11418.555230] LustreError: 349673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788764023 with bad export cookie 7976234130659341374 [11418.557813] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11418.570783] LustreError: 349673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11418.951376] Lustre: server umount lustre-MDT0001 complete [11430.614711] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [11433.465631] Key type lgssc unregistered [11433.859489] LNet: 353826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11433.870115] LNetError: 353826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11433.880881] LNet: Removed LNI 192.168.203.153@tcp [11434.719194] Key type .llcrypt unregistered [11434.721301] Key type ._llcrypt unregistered [11528.275968] Key type ._llcrypt registered [11528.277655] Key type .llcrypt registered [11528.378797] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [11528.698300] Key type .llcrypt unregistered [11528.700414] Key type ._llcrypt unregistered [11616.007976] Key type ._llcrypt registered [11616.011327] Key type .llcrypt registered [11616.150886] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [11616.444772] Key type .llcrypt unregistered [11616.452119] Key type ._llcrypt unregistered [11646.108270] Key type ._llcrypt registered [11646.110500] Key type .llcrypt registered [11646.236090] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [11646.571795] Key type .llcrypt unregistered [11646.579419] Key type ._llcrypt unregistered [11680.464955] Key type ._llcrypt registered [11680.467269] Key type .llcrypt registered [11680.608372] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [11680.859368] Key type .llcrypt unregistered [11680.865070] Key type ._llcrypt unregistered [11757.597356] Key type ._llcrypt registered [11757.599896] Key type .llcrypt registered [11757.726551] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [11757.998835] Key type .llcrypt unregistered [11758.003488] Key type ._llcrypt unregistered [11792.546225] Key type ._llcrypt registered [11792.550337] Key type .llcrypt registered [11792.748622] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [11793.037585] Key type .llcrypt unregistered [11793.043910] Key type ._llcrypt unregistered [11835.274346] Key type ._llcrypt registered [11835.284370] Key type .llcrypt registered [11835.406772] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [11835.823293] Key type .llcrypt unregistered [11835.827824] Key type ._llcrypt unregistered [11869.062905] Key type ._llcrypt registered [11869.066448] Key type .llcrypt registered [11869.195647] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [11869.610399] Key type .llcrypt unregistered [11869.614041] Key type ._llcrypt unregistered [11901.581027] Key type ._llcrypt registered [11901.582865] Key type .llcrypt registered [11901.751828] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [11902.177392] Key type .llcrypt unregistered [11902.184265] Key type ._llcrypt unregistered [11940.962521] Key type ._llcrypt registered [11940.966602] Key type .llcrypt registered [11941.041560] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [11941.216385] Key type .llcrypt unregistered [11941.218566] Key type ._llcrypt unregistered [12004.688631] Key type ._llcrypt registered [12004.693799] Key type .llcrypt registered [12004.807852] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [12005.066518] Key type .llcrypt unregistered [12005.070441] Key type ._llcrypt unregistered [12037.141716] Key type ._llcrypt registered [12037.144880] Key type .llcrypt registered [12037.275473] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [12037.630314] Key type .llcrypt unregistered [12037.632065] Key type ._llcrypt unregistered [12069.070546] Key type ._llcrypt registered [12069.081102] Key type .llcrypt registered [12069.244250] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [12069.468359] Key type .llcrypt unregistered [12069.470536] Key type ._llcrypt unregistered [12123.381269] Key type ._llcrypt registered [12123.384606] Key type .llcrypt registered [12123.469496] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [12137.999510] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [12138.955140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12139.059407] alg: No test for adler32 (adler32-zlib) [12140.036970] Lustre: Lustre: Build Version: 2.17.58_43_g6a42d2c [12140.213378] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [12141.847309] Key type lgssc registered [12143.015141] Lustre: Echo OBD driver; http://www.lustre.org/ [12175.621055] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12175.652249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12176.878471] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12176.912901] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12176.992383] Lustre: tst250fs-MDT0000: new disk, initializing [12177.097201] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12177.140249] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12181.547719] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [12191.606253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12191.738679] Lustre: 374114:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12191.832870] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12191.844209] Lustre: Skipped 1 previous similar message [12191.921729] Lustre: tst250fs-MDT0001: new disk, initializing [12192.010778] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12192.055778] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12192.069294] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12196.445891] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [12201.216178] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12207.405276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12207.662156] Lustre: tst250fs-OST0000: new disk, initializing [12207.674379] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12207.682684] Lustre: 375745:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12207.784564] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12209.303706] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12209.322802] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12209.595428] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12213.977803] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [12224.344315] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12224.483187] Lustre: tst250fs-OST0001: new disk, initializing [12224.486628] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12224.491995] Lustre: 376618:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12224.577191] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12230.610694] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [12231.330018] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12231.340182] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12231.395510] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12235.989724] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12238.520818] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12238.524645] Lustre: Skipped 2 previous similar messages [12257.250299] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12257.263098] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12257.272712] Lustre: Skipped 3 previous similar messages [12257.285671] Lustre: Skipped 3 previous similar messages [12261.701861] Lustre: server umount tst250fs-MDT0000 complete [12265.300686] LustreError: 374105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788764869 with bad export cookie 4178959568602186272 [12265.319698] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12265.327508] LustreError: 374105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12265.717341] Lustre: server umount tst250fs-MDT0001 complete [12279.457933] Lustre: server umount tst250fs-OST0000 complete [12283.359261] Lustre: 372338:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788764872/real 1788764872] req@ffff95aaf9161f80 x1875655780270848/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788764888 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12283.401740] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12283.843741] Lustre: server umount tst250fs-OST0001 complete [12302.735833] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [12310.036602] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [12338.499439] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 03:09:01 (1788764941) [12340.290899] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12359.565622] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [12368.850130] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [12401.006329] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12174 sec ========= 03:10:04 (1788765004) [12403.057475] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:10:06 (1788765006) === [12406.252270] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:10:09 (1788765009) === [12445.666291] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [12448.497269] Key type lgssc unregistered [12448.783851] LNet: 385756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12448.796940] LNetError: 385756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12448.817979] LNet: Removed LNI 192.168.203.153@tcp [12449.650175] Key type .llcrypt unregistered [12449.654519] Key type ._llcrypt unregistered