[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 487325568 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002394] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007312] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010132] LSM: Security Framework initializing [ 0.011050] Yama: becoming mindful. [ 0.012040] SELinux: Initializing. [ 0.013071] *** VALIDATE selinux *** [ 0.021566] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025802] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027123] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029111] *** VALIDATE tmpfs *** [ 0.030457] *** VALIDATE proc *** [ 0.032219] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034274] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037033] Spectre V2 : User space: Vulnerable [ 0.039004] Speculative Store Bypass: Vulnerable [ 0.041650] debug: unmapping init [mem 0xffffffffafc59000-0xffffffffafc60fff] [ 0.043931] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044703] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045029] ... version: 2 [ 0.046012] ... bit width: 48 [ 0.047016] ... generic registers: 4 [ 0.048012] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050013] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.052283] rcu: Hierarchical SRCU implementation. [ 0.054312] smp: Bringing up secondary CPUs ... [ 0.055580] x86: Booting SMP configuration: [ 0.056035] .... node #0, CPUs: #1 #2 #3 [ 0.060012] smp: Brought up 1 node, 4 CPUs [ 0.062018] smpboot: Max logical packages: 1 [ 0.063012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139972] node 0 deferred pages initialised in 74ms [ 0.143153] devtmpfs: initialized [ 0.144224] x86/mm: Memory block size: 128MB [ 0.146821] gcov: version magic: 0x41383552 [ 0.149300] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150073] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152299] pinctrl core: initialized pinctrl subsystem [ 0.154207] [ 0.154668] ************************************************************* [ 0.157015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159012] ** ** [ 0.161015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163010] ** ** [ 0.164010] ** This means that this kernel is built to expose internal ** [ 0.167017] ** IOMMU data structures, which may compromise security on ** [ 0.169011] ** your system. ** [ 0.171012] ** ** [ 0.173014] ** If you see this message and you are not debugging the ** [ 0.175014] ** kernel, report this immediately to your vendor! ** [ 0.177012] ** ** [ 0.180020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182018] ************************************************************* [ 0.185543] NET: Registered protocol family 16 [ 0.187427] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.190057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.193062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.196129] cpuidle: using governor menu [ 0.197975] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.201444] PCI: Using configuration type 1 for base access [ 0.203111] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.213304] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.216020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.220038] cryptd: max_cpu_qlen set to 1000 [ 0.222149] ACPI: Added _OSI(Module Device) [ 0.224022] ACPI: Added _OSI(Processor Device) [ 0.226012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.228016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.233142] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.240401] ACPI: Interpreter enabled [ 0.242139] ACPI: PM: (supports S0 S3 S4 S5) [ 0.244020] ACPI: Using IOAPIC for interrupt routing [ 0.246144] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.249425] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.261085] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.263042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.266018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.270077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.275325] acpiphp: Slot [2] registered [ 0.276112] acpiphp: Slot [5] registered [ 0.277105] acpiphp: Slot [6] registered [ 0.278147] acpiphp: Slot [7] registered [ 0.280161] acpiphp: Slot [8] registered [ 0.282146] acpiphp: Slot [9] registered [ 0.283154] acpiphp: Slot [10] registered [ 0.285150] acpiphp: Slot [3] registered [ 0.286066] acpiphp: Slot [4] registered [ 0.286978] acpiphp: Slot [11] registered [ 0.288101] acpiphp: Slot [12] registered [ 0.290086] acpiphp: Slot [13] registered [ 0.291101] acpiphp: Slot [14] registered [ 0.293214] acpiphp: Slot [15] registered [ 0.295112] acpiphp: Slot [16] registered [ 0.296128] acpiphp: Slot [17] registered [ 0.298154] acpiphp: Slot [18] registered [ 0.299101] acpiphp: Slot [19] registered [ 0.301078] acpiphp: Slot [20] registered [ 0.303127] acpiphp: Slot [21] registered [ 0.304108] acpiphp: Slot [22] registered [ 0.305070] acpiphp: Slot [23] registered [ 0.307114] acpiphp: Slot [24] registered [ 0.309118] acpiphp: Slot [25] registered [ 0.310118] acpiphp: Slot [26] registered [ 0.312113] acpiphp: Slot [27] registered [ 0.314103] acpiphp: Slot [28] registered [ 0.315127] acpiphp: Slot [29] registered [ 0.317097] acpiphp: Slot [30] registered [ 0.318101] acpiphp: Slot [31] registered [ 0.320077] PCI host bridge to bus 0000:00 [ 0.321017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.323022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.325022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.327022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.329022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.332025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.333146] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.336010] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.339434] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.350015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.355534] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.358019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.361019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.364021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.367579] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.370845] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.374047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.377346] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.382014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.395019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.399903] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.406823] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.413024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.419024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.433021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.442497] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.453015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.460018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.475017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.485129] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.490021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.495023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.510025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.519087] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.527026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.531022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.545026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.554701] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.563016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.570016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.587019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.597921] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.604028] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.611020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.624035] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.638311] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.641404] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.643369] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.646381] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.649290] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.655157] iommu: Default domain type: Passthrough [ 0.656541] SCSI subsystem initialized [ 0.657000] ACPI: bus type USB registered [ 0.657000] usbcore: registered new interface driver usbfs [ 0.660121] usbcore: registered new interface driver hub [ 0.662112] usbcore: registered new device driver usb [ 0.664233] pps_core: LinuxPPS API ver. 1 registered [ 0.666017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.670080] PTP clock support registered [ 0.672130] EDAC MC: Ver: 3.0.0 [ 0.674258] PCI: Using ACPI for IRQ routing [ 0.676063] NetLabel: Initializing [ 0.677014] NetLabel: domain hash size = 128 [ 0.678137] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.679239] NetLabel: unlabeled traffic allowed by default [ 0.681251] vgaarb: loaded [ 0.683321] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.685020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.692147] clocksource: Switched to clocksource kvm-clock [ 0.804742] VFS: Disk quotas dquot_6.6.0 [ 0.806524] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.808898] *** VALIDATE ramfs *** [ 0.809728] *** VALIDATE hugetlbfs *** [ 0.811241] pnp: PnP ACPI init [ 0.813842] pnp: PnP ACPI: found 6 devices [ 0.832582] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.836972] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.839198] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.841168] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.843712] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.846405] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.849169] NET: Registered protocol family 2 [ 0.851623] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.856435] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.860255] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.866677] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.870935] TCP: Hash tables configured (established 65536 bind 65536) [ 0.874084] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.877266] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.880280] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.883333] NET: Registered protocol family 1 [ 0.886841] RPC: Registered named UNIX socket transport module. [ 0.889340] RPC: Registered udp transport module. [ 0.891039] RPC: Registered tcp transport module. [ 0.892418] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.895158] NET: Registered protocol family 44 [ 0.897343] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.899201] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.900686] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.902353] PCI: CLS 0 bytes, default 64 [ 0.903555] Unpacking initramfs... [ 2.284764] debug: unmapping init [mem 0xffff9e3ffcc54000-0xffff9e3ffffbffff] [ 2.289148] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.291649] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.294886] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.800409] Initialise system trusted keyrings [ 2.802242] Key type blacklist registered [ 2.805504] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.814291] zbud: loaded [ 2.817532] *** VALIDATE nfs *** [ 2.819091] *** VALIDATE nfs4 *** [ 2.821136] pstore: using deflate compression [ 2.825553] Platform Keyring initialized [ 2.931340] NET: Registered protocol family 38 [ 2.933438] Key type asymmetric registered [ 2.935246] Asymmetric key parser 'x509' registered [ 2.937632] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.940824] io scheduler mq-deadline registered [ 2.942382] io scheduler kyber registered [ 2.943934] io scheduler bfq registered [ 2.946133] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.949717] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.953191] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.956613] ACPI: Power Button [PWRF] [ 2.962231] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.969248] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.984351] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.989559] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.005085] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.034062] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.063146] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.069858] Non-volatile memory driver v1.3 [ 3.071624] Linux agpgart interface v0.103 [ 3.103491] virtio_blk virtio1: [vda] 136552 512-byte logical blocks (69.9 MB/66.7 MiB) [ 3.106633] vda: detected capacity change from 0 to 69914624 [ 3.122251] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.125925] vdb: detected capacity change from 0 to 1073741824 [ 3.139594] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.142802] vdc: detected capacity change from 0 to 2621440000 [ 3.162278] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.165545] vdd: detected capacity change from 0 to 2621440000 [ 3.181840] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.185080] vde: detected capacity change from 0 to 4294967296 [ 3.200552] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.202995] vdf: detected capacity change from 0 to 4294967296 [ 3.209060] libphy: Fixed MDIO Bus: probed [ 3.214470] usbcore: registered new interface driver usbserial_generic [ 3.217555] usbserial: USB Serial support registered for generic [ 3.226847] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.231292] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.233225] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.237255] mousedev: PS/2 mouse device common for all mice [ 3.240755] rtc_cmos 00:05: RTC can wake from S4 [ 3.244063] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.248460] rtc_cmos 00:05: registered as rtc0 [ 3.251231] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.251517] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.257483] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.258034] intel_pstate: CPU model not supported [ 3.264570] hid: raw HID events driver (C) Jiri Kosina [ 3.267305] usbcore: registered new interface driver usbhid [ 3.269509] usbhid: USB HID core driver [ 3.274073] drop_monitor: Initializing network drop monitor service [ 3.276833] Initializing XFRM netlink socket [ 3.279072] NET: Registered protocol family 10 [ 3.282385] Segment Routing with IPv6 [ 3.283874] NET: Registered protocol family 17 [ 3.286288] mpls_gso: MPLS GSO support [ 3.292279] RAS: Correctable Errors collector initialized. [ 3.294744] AVX version of gcm_enc/dec engaged. [ 3.296752] AES CTR mode by8 optimization enabled [ 3.374710] sched_clock: Marking stable (3374678387, 0)->(4313372884, -938694497) [ 3.379070] registered taskstats version 1 [ 3.381598] Loading compiled-in X.509 certificates [ 3.383807] zswap: loaded using pool lzo/zbud [ 3.408925] Key type big_key registered [ 3.423680] Key type encrypted registered [ 3.425471] ima: No TPM chip found, activating TPM-bypass! [ 3.427760] ima: Allocated hash algorithm: sha1 [ 3.429641] ima: No architecture policies found [ 3.431628] evm: Initialising EVM extended attributes: [ 3.433863] evm: security.selinux [ 3.435346] evm: security.ima [ 3.436602] evm: security.capability [ 3.438159] evm: HMAC attrs: 0x1 [ 3.440866] rtc_cmos 00:05: setting system clock to 2026-06-12 02:43:47 UTC (1781232227) [ 3.447785] debug: unmapping init [mem 0xffffffffb0c03000-0xffffffffb0dfffff] [ 3.451176] debug: unmapping init [mem 0xffffffffaf982000-0xffffffffafc58fff] [ 3.460122] Write protecting the kernel read-only data: 28672k [ 3.464108] debug: unmapping init [mem 0xffffffffae003000-0xffffffffae1fffff] [ 3.468089] debug: unmapping init [mem 0xffffffffae914000-0xffffffffae9fffff] [ 3.499353] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.508091] systemd[1]: Detected virtualization kvm. [ 3.510192] systemd[1]: Detected architecture x86-64. [ 3.514276] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.539840] systemd[1]: No hostname configured. [ 3.541737] systemd[1]: Set hostname to . [ 3.543852] random: systemd: uninitialized urandom read (16 bytes read) [ 3.545759] systemd[1]: Initializing machine ID from random generator. [ 3.595896] random: ln: uninitialized urandom read (6 bytes read) [ 3.682958] random: systemd: uninitialized urandom read (16 bytes read) [ 3.685830] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.689818] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.693952] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.344201] device-mapper: uevent: version 1.0.3 [ 4.347485] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.088503] virtio_net virtio0 ens2: renamed from eth0 [ 5.103460] random: fast init done [ 5.145272] scsi host0: ata_piix [ 5.172869] scsi host1: ata_piix [ 5.177872] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.180947] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.842410] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.943487] random: crng init done [ 9.945143] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.333202] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.407457] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.644769] SELinux: Disabled at runtime. [ 11.703931] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.711603] systemd[1]: Detected virtualization kvm. [ 11.713451] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.158743] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.162347] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.167071] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.171413] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.175507] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.183576] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.189365] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket.[ 12.231200] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.586993] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.857870] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.915208] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.001897] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.015309] EDAC sbridge: Ver: 1.1.2 [ 14.557211] Key type dns_resolver registered [ 14.869260] NFS: Registering the id_resolver key type [ 14.871607] Key type id_resolver registered [ 14.873170] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg335-server login: [ 40.211540] libcfs: loading out-of-tree module taints kernel. [ 40.231952] Key type ._llcrypt registered [ 40.233505] Key type .llcrypt registered [ 40.281329] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 51.868072] hrtimer: interrupt took 2625523 ns [ 52.505391] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 54.058326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 54.079158] alg: No test for adler32 (adler32-zlib) [ 55.575784] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 56.639572] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 58.495582] Key type lgssc registered [ 61.139101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 83.420403] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 128.019576] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 142.214176] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 142.259985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 143.616640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 143.665288] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 143.789464] Lustre: lustre-MDT0000: new disk, initializing [ 143.894671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 143.927550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 148.562955] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 163.325698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 163.500825] Lustre: 6481:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 163.586661] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 163.597839] Lustre: Skipped 1 previous similar message [ 163.755604] Lustre: lustre-MDT0001: new disk, initializing [ 163.844412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 163.870712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 163.877306] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 168.599547] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.609421] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 184.303980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 184.648848] Lustre: lustre-OST0000: new disk, initializing [ 184.656754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 184.669143] Lustre: 8385:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 184.769874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.508238] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.587769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 193.608384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 193.747280] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 208.152549] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 208.362982] Lustre: lustre-OST0001: new disk, initializing [ 208.366518] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 208.374146] Lustre: 9441:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 208.450346] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 215.339663] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.619038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 217.633048] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 217.691588] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 228.466359] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 236.430130] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 243.779635] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing check_logdir /tmp/testlogs/ [ 248.875612] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing yml_node [ 253.693906] Lustre: DEBUG MARKER: Client: 2.17.53.74 [ 256.764996] Lustre: DEBUG MARKER: MDS: 2.17.53.74 [ 259.794153] Lustre: DEBUG MARKER: OSS: 2.17.53.74 [ 261.786666] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jun 11 22:48:05 EDT 2026 [ 282.086635] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 283.897174] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 302.056532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 302.068282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 302.085263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 304.615831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.620600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 304.633612] Lustre: Skipped 2 previous similar messages [ 304.648403] Lustre: Skipped 3 previous similar messages [ 309.730294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 314.335150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 314.649398] Lustre: server umount lustre-MDT0000 complete [ 318.620554] LustreError: 7861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232542 with bad export cookie 14920397464401243047 [ 318.626087] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 318.631975] LustreError: 7861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 319.985639] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 319.988481] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 319.994848] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 319.994856] Lustre: Skipped 3 previous similar messages [ 320.011115] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 320.024797] Lustre: Skipped 1 previous similar message [ 324.000266] LustreError: 6487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.000419] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 324.031275] Lustre: Skipped 2 previous similar messages [ 325.047736] Lustre: server umount lustre-MDT0001 complete [ 328.992702] Lustre: server umount lustre-OST0000 complete [ 333.141314] Lustre: server umount lustre-OST0001 complete [ 340.271300] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 350.015722] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 389.876309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 390.238475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 390.315628] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 390.473326] Lustre: lustre-MDT0000: new disk, initializing [ 390.567622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 390.597069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 395.938506] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 408.073760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 408.291744] Lustre: 14962:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 408.379255] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 408.384671] Lustre: Skipped 1 previous similar message [ 408.489866] Lustre: lustre-MDT0001: new disk, initializing [ 408.622795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 408.673046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 408.686582] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 413.771389] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 422.063078] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 426.134560] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 433.695032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 434.042712] Lustre: lustre-OST0000: new disk, initializing [ 434.048714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 434.054811] Lustre: 16706:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 434.212883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 435.677553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 435.697773] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 435.805898] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 441.581689] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 450.205904] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 455.609946] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 455.901512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 460.656514] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 460.860321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 466.400686] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 466.418830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.425243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 466.434517] Lustre: Skipped 1 previous similar message [ 466.444816] Lustre: Skipped 1 previous similar message [ 469.326472] Lustre: server umount lustre-OST0000 complete [ 475.104459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 475.109166] LustreError: Skipped 1 previous similar message [ 475.111928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 479.726816] Lustre: server umount lustre-MDT0000 complete [ 481.775587] LustreError: 16203:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 481.805674] LustreError: 16203:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 483.770593] LustreError: 14955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232707 with bad export cookie 14920397464401244580 [ 483.775696] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 483.783828] LustreError: 14955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 484.181180] Lustre: server umount lustre-MDT0001 complete [ 490.998844] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 22:51:54 (1781232714) [ 510.893843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 511.246741] LustreError: 19432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.330314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 515.615987] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 516.579809] LustreError: 19433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.737663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 529.482034] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 535.288221] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 538.916783] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 540.935038] Lustre: Failing over lustre-MDT0000 [ 541.151028] Lustre: server umount lustre-MDT0000 complete [ 541.665924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 541.671884] LustreError: 20427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.683675] Lustre: Skipped 2 previous similar messages [ 541.702581] LustreError: 20427:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 557.026640] Lustre: 3634:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781232765/real 1781232765] req@ffff9e3f47c8da40 x1867757419396864/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781232781 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 557.063391] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 559.969044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.145447] LustreError: 19429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.196137] LustreError: 19429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 567.264981] LustreError: 3631:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e407f0f92c0 x1867757419399680/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 570.721985] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 570.743486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 570.763630] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 571.885411] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 580.109620] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 586.219423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 586.233883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 586.256972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 586.274158] Lustre: Skipped 3 previous similar messages [ 590.483394] Lustre: server umount lustre-MDT0000 complete [ 594.570465] LustreError: 19414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232818 with bad export cookie 14920397464401245994 [ 594.577144] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 594.587282] LustreError: 19414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 595.024765] Lustre: server umount lustre-MDT0001 complete [ 603.834719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 604.308656] LustreError: 22430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 604.331792] LustreError: 22430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 604.388593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 604.391216] Lustre: Skipped 2 previous similar messages [ 608.737414] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 617.328562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 621.922581] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 628.536764] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 633.070531] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 635.526327] Lustre: Failing over lustre-MDT0000 [ 635.539259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.547924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 635.553105] Lustre: Skipped 2 previous similar messages [ 635.568763] Lustre: Skipped 3 previous similar messages [ 635.834820] Lustre: server umount lustre-MDT0000 complete [ 655.560198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 655.674371] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 656.099780] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 660.221681] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 661.475814] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 661.484254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 661.507132] Lustre: Skipped 1 previous similar message [ 661.518255] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 668.179075] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 678.201064] Lustre: server umount lustre-MDT0000 complete [ 680.930527] LustreError: 22427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 680.965624] LustreError: 22427:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 682.148079] LustreError: 22412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232906 with bad export cookie 14920397464401246932 [ 682.158094] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 682.170163] LustreError: 22412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 682.807459] Lustre: server umount lustre-MDT0001 complete [ 694.475870] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 22:55:17 (1781232917) [ 713.158379] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 724.163762] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 770.913620] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 782.263089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 782.542401] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 782.577776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 782.653664] Lustre: lustre-MDT0000: new disk, initializing [ 782.800260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 782.807155] Lustre: Skipped 2 previous similar messages [ 782.825747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 787.885891] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 799.152513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 799.278315] Lustre: 28429:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 799.530866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 804.358825] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 812.554308] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 816.914980] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 824.364965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 824.709170] Lustre: 30171:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 825.924156] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 825.984492] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 826.071047] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 826.103120] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 831.826445] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 839.992877] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 846.671882] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 872.416054] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 872.418700] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 872.445596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 872.447827] Lustre: Skipped 3 previous similar messages [ 872.465231] Lustre: Skipped 3 previous similar messages [ 875.723155] Lustre: server umount lustre-OST0000 complete [ 881.633030] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 886.435953] Lustre: server umount lustre-MDT0000 complete [ 887.779946] LustreError: 28442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 887.799830] LustreError: 28442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 890.940334] LustreError: 28422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233114 with bad export cookie 14920397464401247338 [ 890.948191] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 890.966841] LustreError: 28422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 891.358146] Lustre: server umount lustre-MDT0001 complete [ 903.198262] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 906.840686] Key type lgssc unregistered [ 907.206756] LNet: 32021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.224459] LNetError: 32021:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.261806] LNet: Removed LNI 192.168.203.135@tcp [ 909.387179] Key type .llcrypt unregistered [ 909.390854] Key type ._llcrypt unregistered [ 935.970366] Key type ._llcrypt registered [ 935.973029] Key type .llcrypt registered [ 936.105680] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 937.152776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 937.175037] alg: No test for adler32 (adler32-zlib) [ 938.378960] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 938.633537] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 940.343532] Key type lgssc registered [ 941.559635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 950.277180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 950.290907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 951.749856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 956.275119] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 960.993846] LustreError: 33418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 961.008347] LustreError: 33418:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 965.015119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 965.522652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 970.697497] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 976.529038] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 979.997080] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 986.771525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 987.283735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 990.391227] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 994.588189] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 996.545934] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1000.433133] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1014.259954] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 23:00:38 (1781233238) [ 1016.302591] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1018.038408] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1025.157855] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 23:00:48 (1781233248) [ 1034.720351] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 23:00:58 (1781233258) [ 1035.781674] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1045.287435] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1045.295068] Lustre: Skipped 2 previous similar messages [ 1047.227754] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 23:01:10 (1781233270) [ 1049.663527] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1049.671341] Lustre: Skipped 2 previous similar messages [ 1059.164462] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1059.176978] Lustre: Skipped 4 previous similar messages [ 1073.811377] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 23:01:37 (1781233297) [ 1076.961259] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1080.800950] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1090.222719] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 23:01:53 (1781233313) [ 1092.312677] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1092.325936] Lustre: Skipped 6 previous similar messages [ 1106.784963] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 23:02:10 (1781233330) [ 1123.808417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1123.815301] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1123.830119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1124.850456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1124.850769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1124.857638] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.887393] Lustre: Skipped 1 previous similar message [ 1129.201922] Lustre: server umount lustre-OST0000 complete [ 1134.569488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1134.580836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.604658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1138.714253] Lustre: server umount lustre-MDT0000 complete [ 1140.193852] LustreError: 33418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1140.219347] LustreError: 33418:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1142.251964] LustreError: 33397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233366 with bad export cookie 4759972270642229811 [ 1142.267874] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1142.664202] Lustre: server umount lustre-MDT0001 complete [ 1153.799398] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 1156.717494] Key type lgssc unregistered [ 1157.075443] LNet: 40931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1157.086517] LNetError: 40931:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1157.109636] LNet: Removed LNI 192.168.203.135@tcp [ 1157.981384] Key type .llcrypt unregistered [ 1157.985117] Key type ._llcrypt unregistered [ 1180.688218] Key type ._llcrypt registered [ 1180.692031] Key type .llcrypt registered [ 1180.823354] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 1196.907443] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 1197.893902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1198.135077] alg: No test for adler32 (adler32-zlib) [ 1199.263105] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 1199.526343] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 1201.191227] Key type lgssc registered [ 1202.373565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1239.664468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1239.698063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1241.045396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1241.110749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1241.194252] Lustre: lustre-MDT0000: new disk, initializing [ 1241.313874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1241.340803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1245.988386] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1256.958233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1257.108413] Lustre: 44051:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1257.192815] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1257.200260] Lustre: Skipped 1 previous similar message [ 1257.327966] Lustre: lustre-MDT0001: new disk, initializing [ 1257.412284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1257.447521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1257.455734] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1261.958747] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1269.690171] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1274.232362] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1281.035730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1281.330170] Lustre: lustre-OST0000: new disk, initializing [ 1281.334610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1281.350578] Lustre: 45796:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1281.457417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1287.892574] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1289.260883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1289.269124] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1289.444495] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1295.857395] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1300.375855] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1300.597701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1305.352406] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1305.595569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1310.179376] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1310.185601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1310.199400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1310.205032] Lustre: Skipped 1 previous similar message [ 1313.883293] Lustre: server umount lustre-OST0000 complete [ 1318.880802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1318.895909] LustreError: Skipped 1 previous similar message [ 1318.900731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1318.921487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1318.924414] Lustre: Skipped 1 previous similar message [ 1320.417962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1320.432396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.877167] Lustre: server umount lustre-MDT0000 complete [ 1328.332864] LustreError: 44044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233552 with bad export cookie 9468070943418453306 [ 1328.340725] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1328.700738] Lustre: server umount lustre-MDT0001 complete [ 1340.285371] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 23:06:03 (1781233563) [ 1347.884380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.500345] LustreError: 48109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1348.527478] LustreError: 48109:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1348.595419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1352.935652] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1353.696833] LustreError: 48110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1358.832839] LustreError: 48109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.175579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1362.780243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1367.397854] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1373.620349] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1378.091769] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1385.374488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1385.780282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1391.523244] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1397.050705] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1401.542046] LustreError: 50296:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1402.668808] Lustre: Modifying parameter general.timeout=1 in log params [ 1403.740467] Lustre: Modifying parameter general.timeout=2 in log params [ 1406.663480] Lustre: Modifying parameter general.timeout=5 in log params [ 1406.667891] Lustre: Skipped 2 previous similar messages [ 1410.760475] Lustre: Modifying parameter general.timeout=9 in log params [ 1410.771541] Lustre: Skipped 3 previous similar messages [ 1419.579370] Lustre: Modifying parameter general.timeout=18 in log params [ 1419.586091] Lustre: Skipped 8 previous similar messages [ 1436.561382] Lustre: Modifying parameter general.timeout=35 in log params [ 1436.567718] Lustre: Skipped 16 previous similar messages [ 1469.297470] Lustre: Modifying parameter general.timeout=72 in log params [ 1469.301913] Lustre: Skipped 36 previous similar messages [ 1535.511478] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 23:09:19 (1781233759) [ 1536.583731] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1536.589503] Lustre: Skipped 58 previous similar messages [ 1538.457299] Lustre: 57245:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1540.218479] Lustre: 57353:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1543.905315] Lustre: 57550:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1543.919443] Lustre: 57550:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1577.441936] Lustre: 59125:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1577.452576] Lustre: 59125:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 2 previous similar messages [ 1577.482162] LustreError: 59125:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1581.541524] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 23:10:05 (1781233805) [ 1594.856280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1594.867554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1594.873085] Lustre: Skipped 3 previous similar messages [ 1599.969944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1599.981778] Lustre: Skipped 2 previous similar messages [ 1600.988961] Lustre: server umount lustre-MDT0000 complete [ 1604.764192] LustreError: 48091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233828 with bad export cookie 9468070943418454188 [ 1604.773376] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1604.776587] LustreError: 48091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1605.080372] Lustre: server umount lustre-MDT0001 complete [ 1619.249954] Lustre: server umount lustre-OST0000 complete [ 1649.788160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1650.160648] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1650.542394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1655.107468] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1664.850797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1664.952964] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1665.000693] Lustre: 61661:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1670.390707] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1676.693725] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1681.154841] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1688.436789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1688.617668] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1688.915406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1688.925031] Lustre: Skipped 1 previous similar message [ 1694.421419] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1700.774512] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1702.453916] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1702.461537] Lustre: Skipped 22 previous similar messages [ 1702.520717] Lustre: 63888:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1710.562600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1710.573619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1710.585484] Lustre: Skipped 2 previous similar messages [ 1710.599028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1715.440182] Lustre: server umount lustre-OST0000 complete [ 1721.314580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1721.331155] Lustre: Skipped 2 previous similar messages [ 1725.311816] Lustre: server umount lustre-MDT0000 complete [ 1726.434664] LustreError: 61667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1726.458092] LustreError: 61667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1728.763219] LustreError: 62384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233952 with bad export cookie 9468070943418457338 [ 1728.772634] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1728.774761] LustreError: 62384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1729.032425] Lustre: server umount lustre-MDT0001 complete [ 1740.071164] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 1742.735915] Key type lgssc unregistered [ 1743.027939] LNet: 65076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1743.041167] LNetError: 65076:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1743.066885] LNet: Removed LNI 192.168.203.135@tcp [ 1743.785910] Key type .llcrypt unregistered [ 1743.789318] Key type ._llcrypt unregistered [ 1767.695220] Key type ._llcrypt registered [ 1767.698643] Key type .llcrypt registered [ 1767.811349] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 1768.891565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1768.922805] alg: No test for adler32 (adler32-zlib) [ 1769.956144] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 1770.189723] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 1771.847190] Key type lgssc registered [ 1772.891618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1779.562495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1779.589730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1781.230053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1785.943449] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1791.456914] LustreError: 66478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1791.476185] LustreError: 66478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1794.568559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1795.084834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1799.119149] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1804.867618] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1808.845579] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1815.341960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1815.939184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1822.256465] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1828.086503] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1836.258905] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1846.246670] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1846.257853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1846.301182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1846.758958] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1846.788469] Lustre: Skipped 1 previous similar message [ 1851.126672] Lustre: server umount lustre-MDT0000 complete [ 1851.872639] LustreError: 66477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1851.902482] LustreError: 66477:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1856.773986] LustreError: 66456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234080 with bad export cookie 14493113775208077291 [ 1856.789414] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1856.789811] LustreError: 66456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1857.005522] LustreError: 66472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1857.032157] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1857.317788] Lustre: server umount lustre-MDT0001 complete [ 1865.019424] Lustre: server umount lustre-OST0000 complete [ 1892.892128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1893.274115] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1893.318113] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1893.609480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1898.134473] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1906.227796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1906.328717] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1906.391773] Lustre: 71006:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1906.951607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1911.295350] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1916.707317] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1920.306555] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1926.558798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1926.700951] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1927.008219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1933.268877] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1940.146579] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1942.338336] Lustre: 73243:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1944.425819] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1952.748864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1952.758135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1952.764687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1952.764695] Lustre: Skipped 2 previous similar messages [ 1952.775157] Lustre: Skipped 1 previous similar message [ 1952.812650] LustreError: Skipped 1 previous similar message [ 1956.481574] Lustre: server umount lustre-OST0000 complete [ 1962.978526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1962.991746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1962.996670] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1962.997355] Lustre: Skipped 1 previous similar message [ 1963.029315] Lustre: Skipped 2 previous similar messages [ 1966.872796] Lustre: server umount lustre-MDT0000 complete [ 1968.107407] LustreError: 71019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1968.129107] LustreError: 71019:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1971.194162] LustreError: 70998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234195 with bad export cookie 14493113775208078131 [ 1971.198932] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1971.211701] LustreError: 70998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1971.734280] Lustre: server umount lustre-MDT0001 complete [ 1983.458570] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 1986.518796] Key type lgssc unregistered [ 1986.889578] LNet: 74438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1986.896899] LNetError: 74438:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1987.944192] LNet: Removed LNI 192.168.203.135@tcp [ 1988.925293] Key type .llcrypt unregistered [ 1988.927839] Key type ._llcrypt unregistered [ 2014.744715] Key type ._llcrypt registered [ 2014.748555] Key type .llcrypt registered [ 2014.870481] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 2015.798222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2015.818476] alg: No test for adler32 (adler32-zlib) [ 2016.972493] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 2017.263242] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 2018.943191] Key type lgssc registered [ 2020.323804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2029.487541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2029.507724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2031.113937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2035.820995] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2041.322505] LustreError: 75838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2041.343447] LustreError: 75838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2046.258030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2046.432790] LustreError: 75839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2046.755604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2052.633148] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2059.518298] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2063.552627] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2071.756140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2072.358730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2079.268078] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2086.382792] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2097.154591] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2098.413616] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2099.720984] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2102.110519] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2102.117761] Lustre: Skipped 1 previous similar message [ 2106.209437] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2106.219685] Lustre: Skipped 3 previous similar messages [ 2114.514757] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2114.525516] Lustre: Skipped 8 previous similar messages [ 2130.634111] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2130.645798] Lustre: Skipped 15 previous similar messages [ 2163.259928] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2163.268334] Lustre: Skipped 32 previous similar messages [ 2205.137409] Lustre: 83606:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2208.191325] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 23:20:31 (1781234431) [ 2209.938103] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2212.203622] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 23:20:35 (1781234435) [ 2220.003469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2220.008241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2220.019689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2220.022196] Lustre: Skipped 1 previous similar message [ 2220.038899] LustreError: Skipped 1 previous similar message [ 2225.125953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2225.133530] Lustre: Skipped 2 previous similar messages [ 2230.240981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2230.258162] Lustre: Skipped 1 previous similar message [ 2231.263486] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2231.425900] Lustre: server umount lustre-OST0000 complete [ 2235.362401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2235.362958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2235.375292] Lustre: Skipped 1 previous similar message [ 2240.480869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2240.491787] Lustre: Skipped 2 previous similar messages [ 2241.523738] Lustre: server umount lustre-MDT0000 complete [ 2245.196515] LustreError: 75818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234469 with bad export cookie 18028488826299174473 [ 2245.214561] LustreError: 75818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2245.214566] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2245.677801] Lustre: server umount lustre-MDT0001 complete [ 2257.402476] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 2260.334717] Key type lgssc unregistered [ 2260.661503] LNet: 84840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2260.673845] LNetError: 84840:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2260.705371] LNet: Removed LNI 192.168.203.135@tcp [ 2261.626245] Key type .llcrypt unregistered [ 2261.631152] Key type ._llcrypt unregistered [ 2266.763845] Key type ._llcrypt registered [ 2266.772236] Key type .llcrypt registered [ 2266.948884] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2271.373233] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules [ 2272.209099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2272.235056] alg: No test for adler32 (adler32-zlib) [ 2273.344720] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 2273.629186] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 2275.351434] Key type lgssc registered [ 2276.732380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2278.366281] LustreError: 86015:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2296.381406] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 2300.488653] LustreError: 86015:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2308.795216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2308.832038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2310.614762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2315.886344] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2320.868951] LustreError: 87015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.894643] LustreError: 87015:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2325.988146] LustreError: 87016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2325.992107] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 23:22:29 (1781234549) [ 2331.116878] LustreError: 87015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.550478] LustreError: 87048:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2332.562993] LustreError: 87048:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2332.584837] LustreError: 87048:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 2333.263039] Lustre: server umount lustre-MDT0000 complete [ 2349.401722] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 2352.546847] Key type lgssc unregistered [ 2352.898143] LNet: 88505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2352.911505] LNetError: 88505:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2352.939715] LNet: Removed LNI 192.168.203.135@tcp [ 2354.034684] Key type .llcrypt unregistered [ 2354.041613] Key type ._llcrypt unregistered [ 2372.236530] Key type ._llcrypt registered [ 2372.237972] Key type .llcrypt registered [ 2372.399123] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 2374.355543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2374.418523] alg: No test for adler32 (adler32-zlib) [ 2375.680751] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 2375.896747] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 2377.615123] Key type lgssc registered [ 2379.389578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2389.987356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2390.010892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2391.838778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2396.483712] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2401.272074] LustreError: 89709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2401.296627] LustreError: 89709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2405.905228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2406.678276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2411.501596] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2417.865359] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2421.863088] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2429.716386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2430.481057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2437.254282] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2443.558096] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2495.410275] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 23:25:19 (1781234719) [ 2497.639323] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2499.674812] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 23:25:23 (1781234723) [ 2508.257943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2508.267940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2508.287904] Lustre: Skipped 2 previous similar messages [ 2511.073535] Lustre: server umount lustre-MDT0000 complete [ 2513.386437] LustreError: 90423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2513.405305] LustreError: 90423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2514.611211] LustreError: 89690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234738 with bad export cookie 14355341468345547412 [ 2514.613192] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2514.621130] LustreError: 89690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2514.990015] Lustre: server umount lustre-MDT0001 complete [ 2529.566946] Lustre: server umount lustre-OST0000 complete [ 2541.442677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2541.910147] LustreError: 93670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2542.126674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2546.766526] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2547.174696] LustreError: 93670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2547.209258] LustreError: 93670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2552.299704] LustreError: 93671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2555.597941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2556.490269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2560.943484] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2566.965437] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2570.997484] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2584.731528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2584.941117] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2584.948667] LustreError: 93658:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2584.962439] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2584.971656] LustreError: 95528:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2584.978688] LustreError: 95528:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2584.982021] LustreError: 95528:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2585.062321] Lustre: server umount lustre-OST0000 complete [ 2585.064962] LustreError: 95528:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2595.192100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2595.432251] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2595.441552] LustreError: 93658:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2595.470191] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2595.489919] LustreError: 95994:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2595.500922] LustreError: 95994:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2595.509405] LustreError: 95994:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2595.607847] Lustre: server umount lustre-OST0000 complete [ 2595.609603] LustreError: 95994:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2607.836268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2608.189196] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2608.199087] Lustre: Found index 0 for lustre-OST0000, updating log [ 2608.205536] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2608.323703] Lustre: lustre-OST0000: new disk, initializing [ 2608.338892] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2608.358859] Lustre: 96529:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2608.902141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2618.455946] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2623.809123] Lustre: Failing over lustre-OST0000 [ 2624.002619] Lustre: server umount lustre-OST0000 complete [ 2632.174604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2632.193950] Lustre: Skipped 2 previous similar messages [ 2632.204481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2632.221679] Lustre: Skipped 1 previous similar message [ 2637.282745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2637.289704] Lustre: Skipped 1 previous similar message [ 2639.474493] Lustre: server umount lustre-MDT0000 complete [ 2642.400754] LustreError: 94384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2642.413565] LustreError: 94384:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2643.888994] LustreError: 95548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234867 with bad export cookie 14355341468345550156 [ 2643.913649] LustreError: 95548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2643.917057] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2644.223867] Lustre: server umount lustre-MDT0001 complete [ 2652.502906] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 23:27:56 (1781234876) [ 2668.421472] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 2678.634643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2679.098672] LustreError: 98841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2679.207566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2684.357877] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2694.153770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.644175] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2703.221102] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2710.383610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2711.285299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2711.295945] Lustre: Skipped 1 previous similar message [ 2711.307858] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2717.684882] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2719.472459] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2719.490440] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2719.522455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2724.590531] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:04 [ 2724.609830] Lustre: lustre-OST0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 2724.610594] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2724.620416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2724.635947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2726.415552] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2726.517734] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2726.644300] Lustre: lustre-OST0001: new disk, initializing [ 2726.648211] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2726.660960] Lustre: 101115:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2733.111406] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2734.738175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2734.747635] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2734.878364] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2741.146071] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2752.939490] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2760.682450] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2760.686467] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2760.710684] Lustre: Skipped 2 previous similar messages [ 2760.724705] Lustre: Skipped 2 previous similar messages [ 2763.765807] Lustre: server umount lustre-MDT0001 complete [ 2765.797323] LustreError: 98836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2765.827051] LustreError: 98836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2772.991489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2773.226367] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2773.233337] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2773.295789] Lustre: 98828:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2775.346383] LustreError: 102669:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2775.363265] LustreError: 102669:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2775.370847] LustreError: 98833:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2775.484932] LustreError: 102680:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.135@tcp: cfg command failed: rc = -17 [ 2775.489822] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2775.489822] [ 2779.135171] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2779.293825] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2791.284457] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 23:30:14 (1781235014) [ 2810.515625] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 2829.330303] Lustre: 104677:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2850.527312] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2854.171341] Lustre: 105812:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2947.556682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2947.567958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2947.583678] Lustre: Skipped 1 previous similar message [ 2947.594929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2947.600494] Lustre: Skipped 1 previous similar message [ 2950.190671] Lustre: server umount lustre-MDT0000 complete [ 2953.194848] LustreError: 98838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2953.214995] LustreError: 98838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2953.450941] LustreError: 98823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235177 with bad export cookie 14355341468345551157 [ 2953.456548] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2953.470230] LustreError: 98823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2953.826812] Lustre: server umount lustre-MDT0001 complete [ 2967.617504] Lustre: server umount lustre-OST0000 complete [ 2981.282571] Lustre: server umount lustre-OST0001 complete [ 2986.595875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2994.919192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3001.517630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3010.368212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3028.944075] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 3037.342984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3037.364524] Lustre: lustre-MDT0000: reset Object Index mappings [ 3037.836502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3037.844520] Lustre: Skipped 2 previous similar messages [ 3041.238616] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3048.661840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3048.681065] Lustre: lustre-MDT0001: reset Object Index mappings [ 3052.962813] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3055.379327] Lustre: 110862:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3061.057170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3062.634599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3066.597957] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3073.018389] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3074.902261] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3080.691429] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3080.713080] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3081.725373] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3091.102916] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3095.438570] Lustre: 112707:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3149.100448] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 23:36:12 (1781235372) [ 3176.936037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3176.945938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3176.967019] Lustre: Skipped 3 previous similar messages [ 3176.974306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3176.980503] Lustre: Skipped 4 previous similar messages [ 3182.273926] Lustre: server umount lustre-MDT0000 complete [ 3183.082566] LustreError: 110481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3183.120434] LustreError: 110481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 3185.788959] LustreError: 109730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235409 with bad export cookie 14355341468345598253 [ 3185.823759] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3185.824897] LustreError: 109730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3188.198317] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.212640] Lustre: Skipped 4 previous similar messages [ 3192.290314] Lustre: server umount lustre-MDT0001 complete [ 3210.208550] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3210.334898] Lustre: server umount lustre-OST0000 complete [ 3219.833926] Lustre: server umount lustre-OST0001 complete [ 3225.692951] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 3235.790346] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 3272.755031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3272.976109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3273.007550] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3273.068528] Lustre: lustre-MDT0000: new disk, initializing [ 3273.138974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3273.144074] Lustre: Skipped 3 previous similar messages [ 3273.158913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3277.268939] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3283.840397] Lustre: server umount lustre-MDT0000 complete [ 3292.684667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3293.063988] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3293.075418] Lustre: 116925:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3293.086614] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3297.354313] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3301.950417] Lustre: server umount lustre-MDT0000 complete [ 3310.798725] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 23:38:54 (1781235534) [ 3312.439413] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3314.656149] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 23:38:57 (1781235537) [ 3332.037563] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 3341.547626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3345.829668] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3354.060276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3354.203785] Lustre: 118938:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3354.258690] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3354.262930] Lustre: Skipped 1 previous similar message [ 3354.375666] Lustre: lustre-MDT0001: new disk, initializing [ 3354.572456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3354.583567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3358.537549] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3362.990270] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3369.303671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3369.605310] Lustre: lustre-OST0000: new disk, initializing [ 3369.611129] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3369.618468] Lustre: 120482:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3371.384781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3371.395163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3371.569268] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3376.051673] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3386.842966] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3387.010818] Lustre: lustre-OST0001: new disk, initializing [ 3387.016732] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3387.031726] Lustre: 121337:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3388.962514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3388.971227] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3389.071294] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3394.471322] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3406.288972] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3410.781507] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3846.758418] Lustre: 127744:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3846.767173] Lustre: 127744:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3846.775208] Lustre: 127744:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3846.781169] Lustre: 127744:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3846.787148] Lustre: 127744:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3846.792700] Lustre: 127744:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4170.001245] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:53:12 (1781236392) [ 4185.578767] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4185.594520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4185.603499] Lustre: Skipped 3 previous similar messages [ 4185.625636] Lustre: Skipped 11 previous similar messages [ 4187.575902] Lustre: server umount lustre-MDT0000 complete [ 4190.690156] LustreError: 122149:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4190.705174] LustreError: 122149:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4191.880452] LustreError: 118932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236415 with bad export cookie 14355341468345667350 [ 4191.891753] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4191.907284] LustreError: 118932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4195.810178] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4195.813864] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4195.838476] Lustre: Skipped 1 previous similar message [ 4195.857969] Lustre: Skipped 1 previous similar message [ 4198.492541] Lustre: server umount lustre-MDT0001 complete [ 4203.280208] LustreError: 128398:0:(ldlm_resource.c:1180:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000400:0x4:0x0].0x0 (ffff9e3f429fc800) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4209.605862] Lustre: server umount lustre-OST0000 complete [ 4220.411160] Lustre: server umount lustre-OST0001 complete [ 4228.394307] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 4240.157552] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 4278.477604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4278.687565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4278.718542] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4278.777907] Lustre: lustre-MDT0000: new disk, initializing [ 4278.834229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4278.838937] Lustre: Skipped 5 previous similar messages [ 4278.850317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4283.236940] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 4294.006727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4294.134373] Lustre: 130773:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4294.202922] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4294.206794] Lustre: Skipped 1 previous similar message [ 4294.299952] Lustre: lustre-MDT0001: new disk, initializing [ 4294.471668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4294.477672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4299.338827] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 4306.817503] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4311.257497] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4318.147984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4318.513510] Lustre: lustre-OST0000: new disk, initializing [ 4318.519055] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4318.537843] Lustre: 132518:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4318.690415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4318.701738] Lustre: Skipped 1 previous similar message [ 4320.498748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4320.522987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4320.748370] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4327.437976] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 4338.835779] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4350.117915] Lustre: lustre-MDD0000: changelog on [ 4353.386529] Lustre: lustre-MDD0001: changelog on [ 6416.049575] Lustre: lustre-MDD0001: changelog off [ 6417.844271] Lustre: 133848:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6418.711605] Lustre: lustre-MDD0000: changelog off [ 6425.059080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6425.060523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6425.076853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6425.076860] Lustre: Skipped 2 previous similar messages [ 6425.101880] LustreError: Skipped 1 previous similar message [ 6430.178218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6430.198702] Lustre: Skipped 2 previous similar messages [ 6435.298499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6435.308733] Lustre: Skipped 1 previous similar message [ 6439.391140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6439.635930] Lustre: server umount lustre-OST0000 complete [ 6445.024966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6445.042669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6445.062262] Lustre: Skipped 1 previous similar message [ 6445.079944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6449.782208] Lustre: server umount lustre-MDT0000 complete [ 6450.657525] LustreError: 138042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6450.680545] LustreError: 138042:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6453.736424] LustreError: 130768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238677 with bad export cookie 14355341468346489990 [ 6453.737930] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6453.756428] LustreError: 130768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6454.382162] Lustre: server umount lustre-MDT0001 complete [ 6467.220492] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 6469.938562] Key type lgssc unregistered [ 6470.169564] LNet: 141561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6470.183472] LNetError: 141561:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6470.209112] LNet: Removed LNI 192.168.203.135@tcp [ 6470.941275] Key type .llcrypt unregistered [ 6470.944457] Key type ._llcrypt unregistered [ 6490.198595] Key type ._llcrypt registered [ 6490.201051] Key type .llcrypt registered [ 6490.326582] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 6506.502662] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 6507.348201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6507.540950] alg: No test for adler32 (adler32-zlib) [ 6508.564983] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 6508.876076] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 6510.600431] Key type lgssc registered [ 6511.687420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6546.582314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6546.599946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6547.896817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6547.941810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6548.036305] Lustre: lustre-MDT0000: new disk, initializing [ 6548.181530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6548.198211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6552.241751] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6562.620854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6562.741772] Lustre: 144728:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6562.798176] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6562.802908] Lustre: Skipped 1 previous similar message [ 6562.871076] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6562.900529] Lustre: lustre-MDT0001: new disk, initializing [ 6562.991197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6563.041209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6563.064591] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6567.615814] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6575.568475] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6579.305867] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6585.814871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6586.177967] Lustre: lustre-OST0000: new disk, initializing [ 6586.184447] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6586.203369] Lustre: 146467:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6586.284841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6591.738339] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6596.132892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6596.142438] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6596.242078] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6598.568042] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6605.312414] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_module obdecho/obdecho [ 6607.349917] LustreError: 147342:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6607.362316] LustreError: 147342:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6609.005491] LustreError: 147439:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6621.672497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6621.688528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6621.698791] Lustre: Skipped 1 previous similar message [ 6626.797818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6626.808117] Lustre: Skipped 4 previous similar messages [ 6627.084463] Lustre: server umount lustre-MDT0000 complete [ 6631.075792] LustreError: 144719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238855 with bad export cookie 14093433866332965886 [ 6631.085030] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6631.090320] LustreError: 144719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6631.394192] Lustre: server umount lustre-MDT0001 complete [ 6645.616812] Lustre: server umount lustre-OST0000 complete [ 6666.924918] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 6678.402611] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 6716.817261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6717.124233] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6717.163396] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6717.301961] Lustre: lustre-MDT0000: new disk, initializing [ 6717.424935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6717.459945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6722.028561] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6732.107872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6732.298037] Lustre: 151121:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6732.387290] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6732.393661] Lustre: Skipped 1 previous similar message [ 6732.464691] Lustre: lustre-MDT0001: new disk, initializing [ 6732.531275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6732.569614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6732.588113] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6737.115112] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6745.015945] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6749.250059] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6756.225926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6756.550704] Lustre: lustre-OST0000: new disk, initializing [ 6756.562555] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6756.576557] Lustre: 152861:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6756.670511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6762.124828] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6765.613071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6765.623202] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6765.685361] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6768.240946] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6771.887119] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6772.097503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6775.914928] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6776.149346] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6780.900091] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6780.901391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6780.907652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6780.922325] Lustre: Skipped 2 previous similar messages [ 6784.333434] Lustre: server umount lustre-OST0000 complete [ 6789.088509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6789.094982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6789.108324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6789.114496] Lustre: Skipped 1 previous similar message [ 6791.140065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6794.234426] Lustre: server umount lustre-MDT0000 complete [ 6797.784679] LustreError: 151113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239021 with bad export cookie 14093433866332966845 [ 6797.786097] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6797.792988] LustreError: 151113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6798.149471] Lustre: server umount lustre-MDT0001 complete [ 6801.651666] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 00:37:05 (1781239025) [ 6803.471821] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6805.152664] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 00:37:08 (1781239028) [ 6812.196853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6812.553618] LustreError: 155128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6812.574764] LustreError: 155128:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6812.634335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6816.640768] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6817.760579] LustreError: 155129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6822.884773] LustreError: 155128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6824.487179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6829.615509] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6835.383626] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6839.748648] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6847.735567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6848.044482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6848.057337] Lustre: Skipped 1 previous similar message [ 6854.390909] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6860.087213] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6867.425081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6867.428288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6867.443343] Lustre: Skipped 2 previous similar messages [ 6867.458235] Lustre: Skipped 3 previous similar messages [ 6872.264844] Lustre: server umount lustre-MDT0000 complete [ 6872.544889] LustreError: 155129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6872.565911] LustreError: 155129:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6875.818360] LustreError: 155109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239099 with bad export cookie 14093433866332967727 [ 6875.824669] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6875.827026] LustreError: 155109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6877.667446] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6877.670397] LustreError: 156238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6877.696397] LustreError: 156238:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6882.171844] Lustre: server umount lustre-MDT0001 complete [ 6887.602824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6898.122409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6898.308651] LustreError: 158632:0:(llog.c:1646:llog_backup()) MGC192.168.203.135@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6898.318874] Lustre: 158632:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.135@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6905.312384] LustreError: 158638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6905.331298] LustreError: 158638:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6922.416850] LustreError: 158638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6922.428184] LustreError: 158638:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 6956.002168] LustreError: 158640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6956.018312] LustreError: 158640:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [ 6959.583258] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6959.590948] Lustre: 158632:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.203.135@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7000.567590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7005.553718] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7005.688940] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7014.296809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7021.026424] LustreError: 156733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7021.032248] LustreError: 142950:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d110ed16 x1867764184770432/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7021.054185] LustreError: 156733:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 45 previous similar messages [ 7025.420633] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7026.664431] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7031.385845] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7033.261890] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7037.605392] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7039.098673] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7061.310668] Lustre: server umount lustre-MDT0000 complete [ 7062.500854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7062.502901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7062.516488] Lustre: Skipped 1 previous similar message [ 7064.857986] LustreError: 157572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239288 with bad export cookie 14093433866332974181 [ 7064.862065] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7064.869686] LustreError: 157572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7065.346261] Lustre: server umount lustre-MDT0001 complete [ 7080.017763] Lustre: server umount lustre-OST0000 complete [ 7100.163837] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 7110.869390] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 7148.279740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7148.686806] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7148.721637] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7148.828797] Lustre: lustre-MDT0000: new disk, initializing [ 7148.961857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7148.975247] Lustre: Skipped 1 previous similar message [ 7149.005179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7153.894885] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7164.906878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7165.039503] Lustre: 163477:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7165.298545] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7170.118936] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7178.824302] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7182.954395] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7190.509196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7190.787238] Lustre: lustre-OST0000: new disk, initializing [ 7190.789650] Lustre: Skipped 1 previous similar message [ 7190.792213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7190.797968] Lustre: Skipped 2 previous similar messages [ 7190.802260] Lustre: 165218:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7197.264206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7197.277116] Lustre: Skipped 1 previous similar message [ 7197.283851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7197.416701] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7197.467227] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7205.527676] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7211.223301] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7211.474777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7216.951056] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7217.294092] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7223.268289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7223.269149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7223.271490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7223.271603] Lustre: Skipped 5 previous similar messages [ 7223.276083] LustreError: Skipped 1 previous similar message [ 7223.287256] Lustre: Skipped 1 previous similar message [ 7225.780309] Lustre: server umount lustre-OST0000 complete [ 7231.968244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7236.863357] Lustre: server umount lustre-MDT0000 complete [ 7238.633347] LustreError: 163489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7238.649886] LustreError: 163489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7241.055109] LustreError: 163468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239465 with bad export cookie 14093433866332991016 [ 7241.063105] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7241.070454] LustreError: 163468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7241.306698] Lustre: server umount lustre-MDT0001 complete [ 7251.337921] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 00:44:34 (1781239474) [ 7260.246022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7267.030346] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7276.762983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7282.229702] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7287.992944] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7292.231495] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7300.638212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7307.226929] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7313.376418] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7324.541754] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 00:45:48 (1781239548) [ 7328.738850] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7328.749275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7328.763708] Lustre: Skipped 2 previous similar messages [ 7328.768326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7328.771462] Lustre: Skipped 4 previous similar messages [ 7334.860390] Lustre: server umount lustre-OST0000 complete [ 7344.957743] Lustre: server umount lustre-MDT0000 complete [ 7349.264235] LustreError: 167465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239573 with bad export cookie 14093433866332991898 [ 7349.274468] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7349.622519] Lustre: server umount lustre-MDT0001 complete [ 7360.515705] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 7363.153529] Key type lgssc unregistered [ 7363.454966] LNet: 170859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7363.467454] LNetError: 170859:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7363.496419] LNet: Removed LNI 192.168.203.135@tcp [ 7364.258306] Key type .llcrypt unregistered [ 7364.263992] Key type ._llcrypt unregistered [ 7382.453345] Key type ._llcrypt registered [ 7382.457382] Key type .llcrypt registered [ 7382.572814] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 7383.768468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7383.859940] alg: No test for adler32 (adler32-zlib) [ 7384.973897] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 7385.218490] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 7386.884674] Key type lgssc registered [ 7388.094834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7395.913576] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7395.952634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7411.487833] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7416.609707] LustreError: 172090:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.135@tcp: failed processing log, type 4: rc = -110 [ 7421.791410] LustreError: 172090:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7421.818749] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7421.837442] LustreError: MGC192.168.203.135@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7421.865567] LustreError: 172090:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7421.885328] LustreError: 172090:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 7421.895252] LustreError: 172090:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 7422.000566] Lustre: server umount lustre-OST0000 complete [ 7422.005552] LustreError: 172090:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7431.356969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7432.849754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7437.168265] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7442.405521] LustreError: 172592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7442.427068] LustreError: 172592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7446.743626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7447.175950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7451.920784] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7457.839449] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7462.637684] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7470.270991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7470.437472] LustreError: 174179:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7470.452459] Lustre: 174179:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.135@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7470.592976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7477.559238] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7484.134797] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7487.976566] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7487.977326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7487.980867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7487.993333] Lustre: Skipped 1 previous similar message [ 7492.685869] Lustre: server umount lustre-MDT0000 complete [ 7493.107871] LustreError: 172591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7493.125581] LustreError: 172591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7496.160229] LustreError: 172573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239720 with bad export cookie 3618737368346857115 [ 7496.171351] LustreError: 172573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7496.172914] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7496.566434] Lustre: server umount lustre-MDT0001 complete [ 7511.683377] Lustre: server umount lustre-OST0000 complete [ 7520.651966] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 7531.012984] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 7572.495854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7572.834465] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7572.887761] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7572.981698] Lustre: lustre-MDT0000: new disk, initializing [ 7573.105471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7573.122240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7577.665837] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7588.604187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7588.777240] Lustre: 177558:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7588.842710] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7588.846614] Lustre: Skipped 1 previous similar message [ 7588.920530] Lustre: lustre-MDT0001: new disk, initializing [ 7589.017905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7589.084186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7589.092243] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7593.648594] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7601.765712] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7606.124088] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7613.750986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7614.117756] Lustre: lustre-OST0000: new disk, initializing [ 7614.120537] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7614.135319] Lustre: 179300:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7614.206442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7620.558322] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7623.767505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7623.775772] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7623.891632] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7628.379444] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7633.757232] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7634.032330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7639.479324] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7639.802646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7644.657467] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7644.686473] Lustre: Skipped 2 previous similar messages [ 7644.699673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7644.717453] Lustre: Skipped 3 previous similar messages [ 7648.640876] Lustre: server umount lustre-OST0000 complete [ 7654.883510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7654.887078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7654.907324] Lustre: Skipped 1 previous similar message [ 7654.918167] Lustre: Skipped 1 previous similar message [ 7659.793087] Lustre: server umount lustre-MDT0000 complete [ 7660.003192] LustreError: 179058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7660.025836] LustreError: 179058:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7664.098605] LustreError: 177571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7664.124184] LustreError: 177571:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7664.181777] LustreError: 177551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239888 with bad export cookie 3618737368346857836 [ 7664.187528] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7664.200726] LustreError: 177551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7664.634566] Lustre: server umount lustre-MDT0001 complete [ 7675.148888] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 00:51:38 (1781239898) [ 7695.339957] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 7698.488196] Key type lgssc unregistered [ 7698.876557] LNet: 182082:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7698.885850] LNetError: 182082:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7698.910184] LNet: Removed LNI 192.168.203.135@tcp [ 7699.916745] Key type .llcrypt unregistered [ 7699.918594] Key type ._llcrypt unregistered [ 7721.145468] Key type ._llcrypt registered [ 7721.147218] Key type .llcrypt registered [ 7721.360262] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 7722.699146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7722.735748] alg: No test for adler32 (adler32-zlib) [ 7723.881659] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 7724.087852] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 7725.807528] Key type lgssc registered [ 7727.215805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7735.750214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7735.768775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7737.223187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7742.362593] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7747.561217] LustreError: 183285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7747.596472] LustreError: 183285:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7752.674930] LustreError: 183286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7753.338042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7753.908330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7759.697968] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7765.613673] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7769.434864] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7777.198081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7777.394870] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7777.399250] LustreError: 184923:0:(llog.c:1687:llog_backup()) MGC192.168.203.135@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7777.406345] Lustre: 184923:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.135@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7777.524918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7783.622332] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7789.754528] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7793.125450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7793.130976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7793.144251] Lustre: Skipped 1 previous similar message [ 7793.156391] Lustre: Skipped 2 previous similar messages [ 7798.241026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7798.556520] Lustre: server umount lustre-MDT0000 complete [ 7802.629399] LustreError: 183265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240026 with bad export cookie 7261240332694780435 [ 7802.633060] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7802.647970] LustreError: 183265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7803.035934] Lustre: server umount lustre-MDT0001 complete [ 7817.388218] Lustre: server umount lustre-OST0000 complete [ 7827.659129] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 7840.964810] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 7880.353902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7880.576321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7880.620412] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7880.743335] Lustre: lustre-MDT0000: new disk, initializing [ 7880.836995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7880.853979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7885.680874] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7897.658222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7897.852593] Lustre: 188303:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7897.918252] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7897.930653] Lustre: Skipped 1 previous similar message [ 7897.934250] LustreError: 189068:0:(llog.c:1687:llog_backup()) MGC192.168.203.135@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7897.944962] LustreError: 189068:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7897.957409] Lustre: 189068:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.135@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7898.007560] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7898.035243] Lustre: Skipped 1 previous similar message [ 7898.101577] Lustre: lustre-MDT0001: new disk, initializing [ 7898.295731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7898.336937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7898.344990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7903.946590] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7911.317725] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7916.542660] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7924.238895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7924.481051] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7924.483924] Lustre: Skipped 1 previous similar message [ 7924.487749] LustreError: 190020:0:(llog.c:1687:llog_backup()) MGC192.168.203.135@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7924.494193] LustreError: 190020:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7924.498276] Lustre: 190020:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.135@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7924.508718] Lustre: 190020:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7924.536892] Lustre: lustre-OST0000: new disk, initializing [ 7924.540452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7924.545537] Lustre: 190045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7924.644991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7931.598495] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7934.059151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7934.088306] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7934.172295] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7939.813296] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7944.685922] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7944.911390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7949.465061] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7949.704206] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7954.920426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7954.922197] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7954.937838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7954.937846] Lustre: Skipped 2 previous similar messages [ 7954.941067] Lustre: Skipped 2 previous similar messages [ 7954.980225] LustreError: Skipped 1 previous similar message [ 7957.948504] Lustre: server umount lustre-OST0000 complete [ 7964.641239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7964.650871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7964.673131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7964.679962] Lustre: Skipped 1 previous similar message [ 7968.730519] Lustre: server umount lustre-MDT0000 complete [ 7970.273174] LustreError: 188314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7970.286939] LustreError: 188314:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7972.824071] LustreError: 188296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240196 with bad export cookie 7261240332694781156 [ 7972.831421] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7972.837087] LustreError: 188296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7973.415195] Lustre: server umount lustre-MDT0001 complete [ 7983.493381] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 00:56:47 (1781240207) [ 8002.315859] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 8012.988542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8013.551117] LustreError: 193009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8013.645711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8018.914345] LustreError: 193010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8018.958212] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8028.131575] LustreError: 193009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8028.153536] LustreError: 193009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8029.280826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8034.128192] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8036.757740] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8043.744890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8051.072443] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8061.091446] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8061.331639] Lustre: lustre-OST0001: new disk, initializing [ 8061.335725] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8061.342799] Lustre: 195268:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8061.454428] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8061.468425] Lustre: Skipped 2 previous similar messages [ 8067.546047] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8068.773585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8068.795510] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8068.897495] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8078.553317] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8087.750399] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8089.993854] Lustre: DEBUG MARKER: ADD OST3 [ 8095.232980] loop: module loaded [ 8095.291826] loop0: detected capacity change from 0 to 204800000 [ 8095.505583] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8095.513653] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8095.523961] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8102.422944] loop0: detected capacity change from 0 to 204800000 [ 8103.244614] Lustre: DEBUG MARKER: STOP OST3 [ 8109.254853] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8109.587348] Lustre: lustre-OST0003: new disk, initializing [ 8109.592711] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8109.600177] Lustre: 197273:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8111.376330] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8111.555262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8111.594404] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8111.647209] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8113.443957] LustreError: 197402:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8114.382141] Lustre: Failing over lustre-OST0003 [ 8114.734476] Lustre: server umount lustre-OST0003 complete [ 8115.946462] LustreError: 194464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8116.712442] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8116.721858] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8116.731887] Lustre: Skipped 2 previous similar messages [ 8118.212149] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8127.471465] Lustre: DEBUG MARKER: START OST3 again [ 8134.969469] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8135.201039] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8135.212417] Lustre: Skipped 1 previous similar message [ 8135.220944] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8136.369491] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8137.328116] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8137.332221] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8142.052519] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8163.897734] Lustre: Failing over lustre-OST0003 [ 8164.088625] Lustre: server umount lustre-OST0003 complete [ 8165.864210] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8165.887222] LustreError: 199455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8165.888401] Lustre: Skipped 1 previous similar message [ 8165.928337] LustreError: 199455:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 8183.265535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8183.277253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8183.306200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8183.311671] Lustre: Skipped 2 previous similar messages [ 8188.855300] Lustre: server umount lustre-MDT0000 complete [ 8192.623028] LustreError: 194484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240416 with bad export cookie 7261240332694782038 [ 8192.627581] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8192.634531] LustreError: 194484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8196.581984] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8196.591201] Lustre: Skipped 5 previous similar messages [ 8199.115635] Lustre: server umount lustre-MDT0001 complete [ 8203.322240] Lustre: server umount lustre-OST0000 complete [ 8207.274769] Lustre: server umount lustre-OST0001 complete [ 8213.776217] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 8223.167870] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 8263.337554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8263.814733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8263.842506] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8263.959385] Lustre: lustre-MDT0000: new disk, initializing [ 8264.104816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8264.143274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8268.799485] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8279.350033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8279.477866] Lustre: 202654:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8279.789300] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8283.761899] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8293.530760] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8298.999772] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8307.641389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8307.886185] Lustre: lustre-OST0000: new disk, initializing [ 8307.892155] Lustre: Skipped 1 previous similar message [ 8307.895282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8307.901598] Lustre: Skipped 2 previous similar messages [ 8307.905400] Lustre: 204396:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8309.942627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8309.958530] Lustre: Skipped 1 previous similar message [ 8309.974731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8310.057556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8314.848107] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8322.994805] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8329.012875] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8329.325568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8334.937337] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8335.229686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8340.971576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8340.974773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8340.976441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8340.976447] Lustre: Skipped 2 previous similar messages [ 8340.992982] Lustre: Skipped 6 previous similar messages [ 8341.024739] LustreError: Skipped 1 previous similar message [ 8343.674787] Lustre: server umount lustre-OST0000 complete [ 8353.995143] Lustre: server umount lustre-MDT0000 complete [ 8356.320483] LustreError: 203514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8356.330951] LustreError: 203514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 8358.558164] LustreError: 202646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240582 with bad export cookie 7261240332694784327 [ 8358.575410] LustreError: 202646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8358.577181] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8359.276124] Lustre: server umount lustre-MDT0001 complete [ 8363.157138] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 01:03:06 (1781240586) [ 8379.855637] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 8390.410555] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 8430.521864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8430.899404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8430.953107] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8431.099143] Lustre: lustre-MDT0000: new disk, initializing [ 8431.197579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8435.852214] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8449.194323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8449.352416] Lustre: 208831:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8449.800251] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8455.299756] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8464.356975] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8469.867419] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8477.601582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8477.853626] Lustre: 210574:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8479.103705] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8484.173436] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8491.559179] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8495.960691] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8496.183772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8500.671491] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8500.887521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8504.802477] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8504.819089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8504.823436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8504.846647] Lustre: Skipped 3 previous similar messages [ 8504.851653] Lustre: Skipped 3 previous similar messages [ 8509.164662] Lustre: server umount lustre-OST0000 complete [ 8519.955118] Lustre: server umount lustre-MDT0000 complete [ 8520.168559] LustreError: 208842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8520.199606] LustreError: 208842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8524.421415] LustreError: 209609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240748 with bad export cookie 7261240332694785209 [ 8524.425162] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8524.441644] LustreError: 209609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8524.784210] Lustre: server umount lustre-MDT0001 complete [ 8534.994240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8535.612932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8535.618748] Lustre: Skipped 5 previous similar messages [ 8539.764638] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8548.955798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8554.090990] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8560.237377] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8564.690672] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8572.424334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8579.301558] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8585.476808] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8599.011654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 8599.022284] LustreError: Skipped 1 previous similar message [ 8599.038151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8599.055449] Lustre: Skipped 2 previous similar messages [ 8599.065587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8599.078634] Lustre: Skipped 3 previous similar messages [ 8602.006367] Lustre: server umount lustre-OST0000 complete [ 8612.511502] Lustre: server umount lustre-MDT0000 complete [ 8616.312239] LustreError: 212623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240840 with bad export cookie 7261240332694786091 [ 8616.321374] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8616.323348] LustreError: 212623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8616.779953] Lustre: server umount lustre-MDT0001 complete [ 8726.705143] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 8729.584723] Key type lgssc unregistered [ 8729.910932] LNet: 215948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8729.921263] LNetError: 215948:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8729.935391] LNet: Removed LNI 192.168.203.135@tcp [ 8730.818188] Key type .llcrypt unregistered [ 8730.821941] Key type ._llcrypt unregistered [ 8758.121369] Key type ._llcrypt registered [ 8758.125325] Key type .llcrypt registered [ 8758.254306] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 8775.575232] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 8776.437765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8776.477902] alg: No test for adler32 (adler32-zlib) [ 8777.703914] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 8778.051661] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 8779.791176] Key type lgssc registered [ 8780.893744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8819.547740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8819.577325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8821.005816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8821.040589] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8821.099532] Lustre: lustre-MDT0000: new disk, initializing [ 8821.170252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8821.186321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8826.491756] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8838.222763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8838.407434] Lustre: 219276:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8838.455691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8838.462934] Lustre: Skipped 1 previous similar message [ 8838.588711] Lustre: lustre-MDT0001: new disk, initializing [ 8838.760326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8838.828312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8838.864500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8844.227770] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8853.718027] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8858.939455] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8868.044185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8868.383264] Lustre: lustre-OST0000: new disk, initializing [ 8868.386642] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8868.392544] Lustre: 221020:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8868.485992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8874.585177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8874.604192] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8874.772986] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8876.435562] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8886.356093] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8892.258302] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8892.517789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8898.347843] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8898.636693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8905.697053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8905.703951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8905.723170] Lustre: Skipped 1 previous similar message [ 8905.746523] Lustre: Skipped 1 previous similar message [ 8907.199049] Lustre: server umount lustre-OST0000 complete [ 8915.428549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8915.442054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8915.471383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8917.937965] Lustre: server umount lustre-MDT0000 complete [ 8922.234440] LustreError: 219267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241146 with bad export cookie 1167502416403341315 [ 8922.253198] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8922.258645] LustreError: 219267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8922.791933] Lustre: server umount lustre-MDT0001 complete [ 8946.130456] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 8959.239594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8960.403619] LustreError: 223791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8960.435657] LustreError: 223791:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8960.488856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8965.600213] LustreError: 223792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8966.813278] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8970.722626] LustreError: 223791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8975.843414] LustreError: 223792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8977.534560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8978.117162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8983.885723] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8987.511374] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8995.656625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8996.162666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9003.649682] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9012.748068] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9012.950426] Lustre: lustre-OST0001: new disk, initializing [ 9012.954265] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9012.960183] Lustre: 226046:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9013.048626] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9019.868088] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9021.055869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9021.077091] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9021.221607] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9032.763152] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9040.689803] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9055.312640] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 01:14:38 (1781241278) [ 9059.815541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9059.828097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9059.856885] Lustre: Skipped 1 previous similar message [ 9059.864792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9059.867776] Lustre: Skipped 1 previous similar message [ 9062.379485] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9062.391254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9062.402695] Lustre: Skipped 2 previous similar messages [ 9062.422972] Lustre: Skipped 3 previous similar messages [ 9063.960547] Lustre: server umount lustre-MDT0000 complete [ 9067.512199] LustreError: 223791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9067.533927] LustreError: 223791:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9067.847894] LustreError: 223772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241291 with bad export cookie 1167502416403342197 [ 9067.849868] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9067.866581] LustreError: 223772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9068.212195] Lustre: server umount lustre-MDT0001 complete [ 9083.098988] Lustre: server umount lustre-OST0000 complete [ 9097.601854] Lustre: server umount lustre-OST0001 complete [ 9104.950920] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 9117.103192] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 9158.516180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9158.867691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9158.905856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9159.053074] Lustre: lustre-MDT0000: new disk, initializing [ 9159.177748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9159.215541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9164.926789] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9179.579117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9179.731372] Lustre: 229942:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9179.771086] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9179.774513] Lustre: Skipped 1 previous similar message [ 9179.846584] Lustre: lustre-MDT0001: new disk, initializing [ 9179.980911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9180.004267] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9184.686855] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9192.728917] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9196.727818] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9203.196796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9203.428433] Lustre: lustre-OST0000: new disk, initializing [ 9203.431789] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9203.437580] Lustre: 231682:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9204.753311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9204.759819] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9204.859364] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9209.148821] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9216.430423] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9220.753533] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9220.991208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9225.414205] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9225.622808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9230.308593] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9230.312715] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9230.320113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9230.326104] Lustre: Skipped 1 previous similar message [ 9233.703688] Lustre: server umount lustre-OST0000 complete [ 9240.546086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9240.552923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9240.566581] Lustre: Skipped 1 previous similar message [ 9240.577876] Lustre: Skipped 2 previous similar messages [ 9243.976193] Lustre: server umount lustre-MDT0000 complete [ 9245.669505] LustreError: 232960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9245.689307] LustreError: 232960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9248.350373] LustreError: 229936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241472 with bad export cookie 1167502416403344332 [ 9248.351348] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9248.371470] LustreError: 229936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9248.824735] Lustre: server umount lustre-MDT0001 complete [ 9258.992370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9259.394394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9259.406334] Lustre: Skipped 2 previous similar messages [ 9263.933838] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9264.614049] LustreError: 233754:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9264.627956] LustreError: 233754:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9273.431345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9279.440226] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9286.435575] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9292.131608] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9299.324531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9305.140901] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9310.669720] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9458.658180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9458.665705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9458.685365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9464.256794] Lustre: server umount lustre-OST0000 complete [ 9468.384995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9468.389630] LustreError: Skipped 1 previous similar message [ 9474.683835] Lustre: server umount lustre-MDT0000 complete [ 9477.089566] LustreError: 235920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9477.099347] LustreError: 235920:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9478.775373] LustreError: 233733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241702 with bad export cookie 1167502416403345214 [ 9478.783668] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9478.801969] LustreError: 233733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9479.198918] Lustre: server umount lustre-MDT0001 complete [ 9579.962527] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 9582.539852] Key type lgssc unregistered [ 9582.804691] LNet: 237001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9582.812087] LNetError: 237001:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9582.829750] LNet: Removed LNI 192.168.203.135@tcp [ 9583.596151] Key type .llcrypt unregistered [ 9583.599337] Key type ._llcrypt unregistered [ 9611.642180] Key type ._llcrypt registered [ 9611.644821] Key type .llcrypt registered [ 9611.755141] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 9626.625448] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 9627.717786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9627.810535] alg: No test for adler32 (adler32-zlib) [ 9628.861628] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 9629.112964] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 9630.792054] Key type lgssc registered [ 9631.941867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9676.943124] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 9686.720564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9686.768340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9688.115641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9688.137573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9688.215301] Lustre: lustre-MDT0000: new disk, initializing [ 9688.286110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9688.300996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9692.275128] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9701.748722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9701.846202] Lustre: 241023:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9701.882491] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9701.885716] Lustre: Skipped 1 previous similar message [ 9701.958977] Lustre: lustre-MDT0001: new disk, initializing [ 9702.057689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9702.093337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9702.112820] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9706.132722] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9711.109340] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9717.220653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9717.499840] Lustre: lustre-OST0000: new disk, initializing [ 9717.503198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9717.511047] Lustre: 242618:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9717.594988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9718.313938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9718.326630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9718.449273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9723.418144] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9733.466220] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9733.647855] Lustre: lustre-OST0001: new disk, initializing [ 9733.652921] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9733.662535] Lustre: 243472:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9733.766156] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9740.185268] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9743.018492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9743.027975] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9743.199621] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9752.312469] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9760.746376] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9774.052297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9774.057313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9774.067230] Lustre: Skipped 1 previous similar message [ 9774.081642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9779.228987] Lustre: server umount lustre-MDT0000 complete [ 9782.310819] LustreError: 241809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242006 with bad export cookie 17756379003670011183 [ 9782.325381] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9782.326416] LustreError: 241809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9782.708579] Lustre: server umount lustre-MDT0001 complete [ 9796.378840] Lustre: server umount lustre-OST0000 complete [ 9810.163768] Lustre: server umount lustre-OST0001 complete [ 9823.584830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9833.441141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9842.407910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9852.580175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9874.098588] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 9884.114337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9884.152947] Lustre: lustre-MDT0000: reset Object Index mappings [ 9884.685510] LustreError: 248451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9884.710583] LustreError: 248451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9884.777526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9889.080915] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9889.766908] LustreError: 248452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9894.881830] LustreError: 248451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9897.864711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9897.923579] Lustre: lustre-MDT0001: reset Object Index mappings [ 9898.473038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9902.995696] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9906.213818] Lustre: 249558:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9912.734346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9919.166955] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9923.314748] LustreError: 249913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9926.951951] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9927.181951] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9927.188179] Lustre: Skipped 1 previous similar message [ 9929.282826] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9929.284878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9933.365303] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9940.805496] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9944.941836] Lustre: 251400:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9952.129558] Lustre: server umount lustre-MDT0000 complete [ 9954.788996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9954.794372] LustreError: 248448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9954.814683] Lustre: Skipped 4 previous similar messages [ 9954.863097] LustreError: 248448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9956.183277] LustreError: 249177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242180 with bad export cookie 17756379003670017973 [ 9956.191018] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9956.197942] LustreError: 249177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9956.495395] Lustre: server umount lustre-MDT0001 complete [ 9972.085924] Lustre: server umount lustre-OST0000 complete [ 9975.261563] Lustre: 238544:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781242183/real 1781242183] req@ffff9e405de78000 x1867767457375488/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781242199 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9975.286622] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9975.298599] Lustre: Skipped 1 previous similar message [ 9976.147122] Lustre: server umount lustre-OST0001 complete [ 9994.352614] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 01:30:17 (1781242217) [10010.572448] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [10021.411113] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [10063.831704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10064.105531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10064.136423] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10064.304176] Lustre: lustre-MDT0000: new disk, initializing [10064.412797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10064.443890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10069.488498] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10081.118233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10081.280210] Lustre: 255619:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10081.338334] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10081.343325] Lustre: Skipped 1 previous similar message [10081.406671] Lustre: lustre-MDT0001: new disk, initializing [10081.544126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10081.563738] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10086.479353] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10094.574767] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10098.966395] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10106.630630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10106.963428] Lustre: lustre-OST0000: new disk, initializing [10106.967979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10106.976291] Lustre: 257360:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10108.271048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10108.282841] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10108.481424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10113.736864] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10123.081662] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10128.091730] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10128.440977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10133.030576] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10133.240404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10139.106670] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10139.107265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10139.119414] LustreError: Skipped 1 previous similar message [10139.120102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10139.120106] Lustre: Skipped 3 previous similar messages [10139.172100] Lustre: Skipped 1 previous similar message [10144.236831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10144.250682] Lustre: Skipped 2 previous similar messages [10149.344366] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10149.359251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10149.372671] Lustre: Skipped 1 previous similar message [10149.595713] Lustre: server umount lustre-OST0000 complete [10153.440113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10153.446249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10153.459210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10159.587118] LustreError: 255630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10159.612649] LustreError: 255630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10159.674417] Lustre: server umount lustre-MDT0000 complete [10163.798362] LustreError: 255610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242387 with bad export cookie 17756379003670020066 [10163.801359] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10163.813269] LustreError: 255610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10164.216718] Lustre: server umount lustre-MDT0001 complete [10185.994587] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [10195.735762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10196.196911] LustreError: 260146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10196.259715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10196.265434] Lustre: Skipped 2 previous similar messages [10200.308567] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10208.690334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10213.501692] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10216.413792] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10223.160606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10229.306156] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10237.762780] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10237.975079] Lustre: lustre-OST0001: new disk, initializing [10237.985332] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10237.996358] Lustre: 262403:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10239.711260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10239.721879] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10239.842342] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10244.505983] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10254.676945] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10258.640015] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10264.150842] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10264.296823] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10272.740625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10272.756774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10272.757718] Lustre: Skipped 3 previous similar messages [10272.771862] Lustre: Skipped 5 previous similar messages [10277.528157] Lustre: server umount lustre-MDT0000 complete [10277.860107] LustreError: 260146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10277.880692] LustreError: 260146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10281.081201] LustreError: 260129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242505 with bad export cookie 17756379003670020948 [10281.091064] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10281.112611] LustreError: 260129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10281.397378] Lustre: server umount lustre-MDT0001 complete [10295.491612] Lustre: server umount lustre-OST0000 complete [10309.360989] Lustre: server umount lustre-OST0001 complete [10327.225491] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [10336.890146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10337.297173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10337.301277] Lustre: Skipped 3 previous similar messages [10341.264522] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10342.368824] LustreError: 265425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10342.389903] LustreError: 265425:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10349.093562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10353.489445] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10356.377803] Lustre: 266532:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10363.167927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10369.864673] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10378.190592] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10382.587987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10382.598573] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10384.574628] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10392.562627] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10396.543486] Lustre: 268376:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10402.361793] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10402.499297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10413.239356] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 01:37:16 (1781242636) [10428.903320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10428.909459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10428.919387] Lustre: Skipped 3 previous similar messages [10428.932404] Lustre: Skipped 3 previous similar messages [10430.563429] Lustre: server umount lustre-MDT0000 complete [10434.144754] LustreError: 265407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242658 with bad export cookie 17756379003670027500 [10434.149241] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10434.162522] LustreError: 265407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10440.582431] Lustre: server umount lustre-MDT0001 complete [10450.465750] Lustre: server umount lustre-OST0000 complete [10454.027866] Lustre: server umount lustre-OST0001 complete [10460.098748] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [10470.430316] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [10519.969505] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [10529.878639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10530.165729] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10530.216864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10530.328806] Lustre: lustre-MDT0000: new disk, initializing [10530.459087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10535.037554] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10545.857464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10545.964737] Lustre: 272564:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10546.239221] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10550.947966] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10563.081724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10563.339377] Lustre: 274163:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10565.178970] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10569.976731] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10580.737520] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10580.946959] Lustre: 275015:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10582.086418] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10582.102495] Lustre: Skipped 1 previous similar message [10582.174593] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10586.718388] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10596.702328] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10600.335838] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10600.341443] Lustre: Skipped 1 previous similar message [10616.698919] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 01:40:40 (1781242840) [10621.053874] Lustre: server umount lustre-MDT0000 complete [10622.946623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10622.950184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10622.972918] Lustre: Skipped 3 previous similar messages [10622.991304] LustreError: 272575:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10623.025689] LustreError: 272575:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [10625.371229] LustreError: 272554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242849 with bad export cookie 17756379003670035144 [10625.378996] LustreError: 272554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10625.384893] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10625.878522] Lustre: server umount lustre-MDT0001 complete [10640.848657] Lustre: server umount lustre-OST0000 complete [10644.576082] Lustre: 238546:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781242852/real 1781242852] req@ffff9e406663cf00 x1867767457631744/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781242868 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10645.154589] Lustre: server umount lustre-OST0001 complete [10653.948526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10654.427364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10654.434794] Lustre: Skipped 7 previous similar messages [10658.745248] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10666.345660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10671.687512] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10677.364116] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10681.747658] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10688.334135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10693.860459] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10698.709633] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10713.393568] loop1: detected capacity change from 0 to 204800000 [10713.624121] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10713.637068] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10713.648049] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10718.507285] loop1: detected capacity change from 0 to 204800000 [10723.413897] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10723.536890] LustreError: 277234:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs27481: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10723.551291] LustreError: 277234:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs27481 [10723.559289] LustreError: 277234:0:(mgs_handler.c:517:mgs_target_reg()) fs27481-MDT0001: Error checking target index: rc = -13 [10723.573761] LustreError: fs27481-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10723.583901] LustreError: 280622:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [10723.593470] LustreError: 280622:0:(tgt_mount.c:2084:server_put_super()) no obd fs27481-MDT0001 [10723.602044] LustreError: 280622:0:(tgt_mount.c:116:server_deregister_mount()) fs27481-MDT0001 not registered [10723.636558] Lustre: server umount fs27481-MDT0001 complete [10723.649261] LustreError: 280622:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10730.792481] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10730.862767] Lustre: Setting parameter fs27481-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs27481-MDT0001 [10730.908807] Lustre: srv-fs27481-MDT0001: No data found on store. Initialize space. [10730.912589] Lustre: Skipped 4 previous similar messages [10731.021596] Lustre: fs27481-MDT0001: new disk, initializing [10731.027785] Lustre: Skipped 3 previous similar messages [10731.183695] Lustre: srv-fs27481-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10731.192769] Lustre: 281088:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27481-MDT0001-mdtlov: can't create new llog: rc = -115 [10731.205561] LustreError: 281088:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27481-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10735.045175] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10741.535636] Lustre: server umount fs27481-MDT0001 complete [10753.558629] loop1: detected capacity change from 0 to 204800000 [10758.548110] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10758.829339] Lustre: 282638:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27481-MDT0001-mdtlov: can't create new llog: rc = -115 [10758.843577] LustreError: 282638:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27481-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10763.260309] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10767.921699] Lustre: server umount fs27481-MDT0001 complete [10776.048523] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10779.619616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10779.635086] Lustre: Skipped 4 previous similar messages [10779.643781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10779.648034] Lustre: Skipped 2 previous similar messages [10782.677730] Lustre: server umount lustre-MDT0000 complete [10790.724777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10790.796856] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10795.112319] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10796.542806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10802.588742] loop1: detected capacity change from 0 to 204800000 [10807.291205] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10807.793103] Lustre: 285048:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27481-MDT0001-mdtlov: can't create new llog: rc = -115 [10807.806159] LustreError: 285048:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27481-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10811.441504] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10815.652646] Lustre: server umount fs27481-MDT0001 complete [10835.424541] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10835.434638] LustreError: Skipped 1 previous similar message [10847.728541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10847.735301] Lustre: Skipped 11 previous similar messages [10849.247127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10849.426423] Lustre: server umount lustre-OST0000 complete [10852.834328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10852.844224] LustreError: Skipped 1 previous similar message [10858.916504] Lustre: server umount lustre-MDT0000 complete [10861.931749] LustreError: 277225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243085 with bad export cookie 17756379003670037832 [10861.932971] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10861.942550] LustreError: 277225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10862.135859] Lustre: server umount lustre-MDT0001 complete [10870.052520] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [10872.626146] Key type lgssc unregistered [10872.905650] LNet: 287359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10872.910114] LNetError: 287359:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10872.935772] LNet: Removed LNI 192.168.203.135@tcp [10873.648128] Key type .llcrypt unregistered [10873.651763] Key type ._llcrypt unregistered [10888.683510] Key type ._llcrypt registered [10888.686749] Key type .llcrypt registered [10888.758772] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [10889.391579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10889.436066] alg: No test for adler32 (adler32-zlib) [10890.404687] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [10890.592975] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [10892.231317] Key type lgssc registered [10893.192266] Lustre: Echo OBD driver; http://www.lustre.org/ [10898.991589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10899.007330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10900.418880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10903.800270] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10910.430723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10910.810926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10913.887636] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10918.397317] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10921.584264] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10927.062692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10927.391564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10932.190501] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10935.588071] LustreError: 290196:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10935.610626] LustreError: 290196:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10936.613402] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10940.903497] LustreError: 290705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10945.006187] LustreError: 290197:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10945.029415] LustreError: 290197:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10948.071251] LustreError: 290198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10948.098225] LustreError: 290198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10953.185884] LustreError: 290196:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10953.197829] LustreError: 290196:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10960.902320] loop1: detected capacity change from 0 to 204800000 [10961.017882] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10961.030834] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10961.039873] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10963.431100] LustreError: 290196:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10963.445203] LustreError: 290196:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10964.941448] loop1: detected capacity change from 0 to 204800000 [10969.185847] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10969.334151] Lustre: fs21064-OST0000: new disk, initializing [10969.336817] Lustre: srv-fs21064-OST0000: No data found on store. Initialize space. [10969.341740] Lustre: 292009:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10969.415423] Lustre: fs21064-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10974.730523] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10979.504334] LustreError: 290197:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10979.531478] LustreError: 290197:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [10983.000417] Lustre: server umount fs21064-OST0000 complete [10993.403930] loop1: detected capacity change from 0 to 204800000 [10998.609165] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10998.860173] Lustre: fs21064-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11004.531791] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11012.654670] loop2: detected capacity change from 0 to 204800000 [11012.911687] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11012.919912] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11012.929608] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11014.645680] LustreError: 290198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11014.672926] LustreError: 290198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [11017.662901] loop2: detected capacity change from 0 to 204800000 [11019.752842] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11022.027808] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11022.108555] LustreError: fs21064-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11022.116503] LustreError: fs21064-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11022.122552] LustreError: 295015:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [11022.128952] LustreError: 295015:0:(tgt_mount.c:2084:server_put_super()) no obd fs21064-OST0001 [11022.133259] LustreError: 295015:0:(tgt_mount.c:116:server_deregister_mount()) fs21064-OST0001 not registered [11022.162281] Lustre: server umount fs21064-OST0001 complete [11022.165096] LustreError: 295015:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11024.868394] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11030.800942] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11030.931443] Lustre: fs21064-OST0001: new disk, initializing [11030.934674] Lustre: srv-fs21064-OST0001: No data found on store. Initialize space. [11030.941299] Lustre: 295565:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11031.023843] Lustre: fs21064-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11036.124693] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11043.494347] Lustre: server umount fs21064-OST0000 complete [11050.580108] Lustre: server umount fs21064-OST0001 complete [11060.628550] loop1: detected capacity change from 0 to 204800000 [11065.220252] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11065.393548] Lustre: fs21064-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11070.182146] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11075.473832] loop2: detected capacity change from 0 to 204800000 [11079.588234] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11081.194288] LustreError: 290198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11081.219799] LustreError: 290198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [11085.210494] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11093.125619] Lustre: server umount fs21064-OST0000 complete [11100.355376] Lustre: server umount fs21064-OST0001 complete [11109.856914] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11109.862843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11109.875450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11111.908378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11111.916400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11111.930103] Lustre: Skipped 1 previous similar message [11115.843705] Lustre: server umount lustre-OST0000 complete [11120.615100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11120.625502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11120.643939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11125.163789] LustreError: 300680:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x5:0x0].0x0 (ffff9e3f4516a700) refcount nonzero (2) after lock cleanup; forcing cleanup. [11125.277789] Lustre: server umount lustre-MDT0000 complete [11128.507601] LustreError: 288574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243352 with bad export cookie 17807991792532537028 [11128.515575] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11128.527243] LustreError: 288574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11128.919211] Lustre: server umount lustre-MDT0001 complete [11138.942866] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [11141.063994] Key type lgssc unregistered [11141.273669] LNet: 301412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11141.283539] LNetError: 301412:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11141.300521] LNet: Removed LNI 192.168.203.135@tcp [11142.025153] Key type .llcrypt unregistered [11142.027266] Key type ._llcrypt unregistered [11168.128910] Key type ._llcrypt registered [11168.131329] Key type .llcrypt registered [11168.216438] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [11181.611654] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [11182.210481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11182.233093] alg: No test for adler32 (adler32-zlib) [11183.240609] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [11183.387619] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [11185.031171] Key type lgssc registered [11185.809788] Lustre: Echo OBD driver; http://www.lustre.org/ [11211.771107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11211.812992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11213.013822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11213.042611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11213.114093] Lustre: lustre-MDT0000: new disk, initializing [11213.170661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11213.187702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11216.251341] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11223.607453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11223.673874] Lustre: 305238:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11223.717835] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11223.721598] Lustre: Skipped 1 previous similar message [11223.784426] Lustre: lustre-MDT0001: new disk, initializing [11223.851501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11223.878424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11223.890425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11226.734261] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11232.298513] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11234.837668] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11239.202897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11239.412367] Lustre: lustre-OST0000: new disk, initializing [11239.415889] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11239.422826] Lustre: 306979:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11239.489602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11240.041443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11240.048203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11240.139598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11243.603264] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11249.197883] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11250.681879] Lustre: Mounted lustre-client [11251.400717] Lustre: Unmounted lustre-client [11260.902386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11260.903564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11260.908374] LustreError: Skipped 1 previous similar message [11260.909617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11260.919414] Lustre: Skipped 1 previous similar message [11260.930794] Lustre: Skipped 1 previous similar message [11266.018429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11266.025706] Lustre: Skipped 1 previous similar message [11271.137096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11271.143852] Lustre: Skipped 1 previous similar message [11272.159138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11272.257977] Lustre: server umount lustre-OST0000 complete [11275.232469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11275.237798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11275.245243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11276.259178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11281.019892] Lustre: server umount lustre-MDT0000 complete [11283.576059] LustreError: 305229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243507 with bad export cookie 8260899519339024735 [11283.579944] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11283.581533] LustreError: 305229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11283.744771] Lustre: server umount lustre-MDT0001 complete [11290.908201] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [11292.692407] Key type lgssc unregistered [11302.112742] LNet: 309011:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11302.122826] LNetError: 309011:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11379.942101] LNet: Removed LNI 192.168.203.135@tcp [11380.503377] Key type .llcrypt unregistered [11380.505531] Key type ._llcrypt unregistered [11391.478130] Key type ._llcrypt registered [11391.480271] Key type .llcrypt registered [11391.546440] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [11402.464118] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [11403.072959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11403.105492] alg: No test for adler32 (adler32-zlib) [11404.032970] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [11404.157324] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [11405.767148] Key type lgssc registered [11406.665857] Lustre: Echo OBD driver; http://www.lustre.org/ [11424.856673] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 01:54:09 (1781243649) [11428.850495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11428.868395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11430.011731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11430.033227] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11430.102563] Lustre: lustre-MDT0000: new disk, initializing [11430.154730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11430.176726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11432.839496] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11440.039836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11440.111938] Lustre: 312174:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11440.154881] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11440.158705] Lustre: Skipped 1 previous similar message [11440.214372] Lustre: lustre-MDT0001: new disk, initializing [11440.279304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11440.316105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11440.324636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11443.090128] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11448.340441] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11450.552926] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11455.054607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11455.240995] Lustre: lustre-OST0000: new disk, initializing [11455.243867] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11455.247577] Lustre: 313914:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11455.303957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11458.871430] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11460.619507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11460.627756] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11460.692469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11463.870367] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11470.819353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11470.821701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11470.824391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11470.831942] Lustre: Skipped 1 previous similar message [11472.319863] Lustre: server umount lustre-OST0000 complete [11475.937981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11475.944382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11475.948048] Lustre: Skipped 1 previous similar message [11475.952675] Lustre: Skipped 1 previous similar message [11476.558161] Lustre: server umount lustre-MDT0000 complete [11478.777946] LustreError: 312165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243702 with bad export cookie 16440930873454608279 [11478.786447] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11478.789581] LustreError: 312165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11478.969393] Lustre: server umount lustre-MDT0001 complete [11485.259792] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [11487.567275] Key type lgssc unregistered [11487.825331] LNet: 315542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11487.829438] LNetError: 315542:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11487.842459] LNet: Removed LNI 192.168.203.135@tcp [11488.595937] Key type .llcrypt unregistered [11488.599671] Key type ._llcrypt unregistered [11523.390455] Key type ._llcrypt registered [11523.392974] Key type .llcrypt registered [11523.562685] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [11525.626829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11525.674292] alg: No test for adler32 (adler32-zlib) [11526.656929] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [11527.011404] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [11528.703176] Key type lgssc registered [11529.724093] Lustre: Echo OBD driver; http://www.lustre.org/ [11540.881652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11540.942481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11542.751650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11548.073870] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11552.237408] LustreError: 316950:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11552.269028] LustreError: 316950:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11557.347117] LustreError: 316951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11558.309876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11558.885106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11563.535210] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11569.918580] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11574.543239] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11583.438592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11583.917563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11590.203329] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11595.452838] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11612.654832] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11614.284751] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11618.505731] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (stopping) [11618.791190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11618.800817] Lustre: Skipped 2 previous similar messages [11623.600557] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (stopping) [11623.606935] Lustre: Skipped 3 previous similar messages [11623.956932] LustreError: 316946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11623.979688] LustreError: 316946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11624.100976] Lustre: server umount lustre-MDT0000 complete [11627.562277] LustreError: 316932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243851 with bad export cookie 13679249739173306511 [11627.566102] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11627.572133] LustreError: 316932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11628.740803] LustreError: 318718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11628.766098] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.35@tcp (stopping) [11629.032846] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11632.097028] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11632.105426] Lustre: Skipped 1 previous similar message [11633.885440] Lustre: server umount lustre-MDT0001 complete [11638.007499] Lustre: server umount lustre-OST0000 complete [11646.666680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11650.624103] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11653.427414] Lustre: 320702:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.135@tcp [11655.391309] Lustre: server umount MGS complete [11663.153789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11663.552177] LustreError: 321273:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11663.561921] LustreError: 321273:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11663.622690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11667.845423] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11673.066846] LustreError: 321274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11673.078346] LustreError: 321274:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11676.100809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11676.491353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11680.793547] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11689.126282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11689.440656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11695.211379] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11700.759099] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11702.403263] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11706.209907] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11711.003642] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [11717.088616] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11717.101557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11717.118907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11719.149089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11719.156786] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11721.402280] Lustre: server umount lustre-OST0000 complete [11727.842601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11727.858225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11727.873279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11727.875811] Lustre: Skipped 2 previous similar messages [11731.130833] Lustre: server umount lustre-MDT0000 complete [11734.497425] LustreError: 321274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11734.519684] LustreError: 321274:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11734.620436] LustreError: 321253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243958 with bad export cookie 13679249739173307351 [11734.628148] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11734.634423] LustreError: 321253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11735.219662] Lustre: server umount lustre-MDT0001 complete [11746.421931] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [11749.217055] Key type lgssc unregistered [11749.595648] LNet: 324449:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11749.613793] LNetError: 324449:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11750.696258] LNet: Removed LNI 192.168.203.135@tcp [11751.556317] Key type .llcrypt unregistered [11751.559212] Key type ._llcrypt unregistered [11770.434351] Key type ._llcrypt registered [11770.440346] Key type .llcrypt registered [11770.615983] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [11787.138537] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [11788.578461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11788.774975] alg: No test for adler32 (adler32-zlib) [11789.923094] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [11790.129533] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [11791.815218] Key type lgssc registered [11793.036377] Lustre: Echo OBD driver; http://www.lustre.org/ [11832.988839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11833.009791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11834.350963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11834.389300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11834.511143] Lustre: lustre-MDT0000: new disk, initializing [11834.640629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11834.672849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11839.473490] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11851.384954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11851.529452] Lustre: 327578:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11851.609936] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11851.612940] Lustre: Skipped 1 previous similar message [11851.744524] Lustre: lustre-MDT0001: new disk, initializing [11851.849915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11851.908379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11851.928307] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11857.074687] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11865.558805] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11870.161903] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11878.711211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11879.045328] Lustre: lustre-OST0000: new disk, initializing [11879.048837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11879.057170] Lustre: 329334:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11879.185622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11884.644130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11884.658518] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11884.772948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11886.459359] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11894.709866] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11900.003615] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11900.267722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11905.319823] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11905.526490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11910.626640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11910.631479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11910.643339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11910.651052] Lustre: Skipped 1 previous similar message [11914.789078] Lustre: server umount lustre-OST0000 complete [11920.865484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11920.892414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11920.904204] Lustre: Skipped 2 previous similar messages [11925.523736] Lustre: server umount lustre-MDT0000 complete [11930.382397] LustreError: 327571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244154 with bad export cookie 14032502236987024471 [11930.393891] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11930.402968] LustreError: 327571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11930.851735] Lustre: server umount lustre-MDT0001 complete [11942.126201] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 02:02:45 (1781244165) [11959.833170] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [11970.347446] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [12007.078421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12007.456794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12007.484131] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12007.580709] Lustre: lustre-MDT0000: new disk, initializing [12007.713470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12007.740641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12013.140967] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12025.209615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12025.357782] Lustre: 333914:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12025.437783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12025.441564] Lustre: Skipped 1 previous similar message [12025.548326] Lustre: lustre-MDT0001: new disk, initializing [12025.672454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12025.712331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12025.726575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12030.520196] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12039.094889] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12044.460263] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12052.593216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12052.945574] Lustre: lustre-OST0000: new disk, initializing [12052.951231] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12052.958040] Lustre: 335657:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12053.126193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12060.263537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12060.277486] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12060.345422] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12060.399825] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12068.688173] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12106.724899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12106.731586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12106.739739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12106.749082] Lustre: Skipped 2 previous similar messages [12110.247621] Lustre: server umount lustre-OST0000 complete [12116.963936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12116.976048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12116.999604] Lustre: Skipped 1 previous similar message [12117.023703] Lustre: Skipped 2 previous similar messages [12121.491978] Lustre: server umount lustre-MDT0000 complete [12122.081190] LustreError: 336363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12122.103575] LustreError: 336363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [12125.906133] LustreError: 333904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244349 with bad export cookie 14032502236987025353 [12125.916063] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12125.924275] LustreError: 333904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12126.301405] Lustre: server umount lustre-MDT0001 complete [12139.272234] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12142.964991] Key type lgssc unregistered [12143.326550] LNet: 337694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12143.339261] LNetError: 337694:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12144.373225] LNet: Removed LNI 192.168.203.135@tcp [12145.526195] Key type .llcrypt unregistered [12145.529923] Key type ._llcrypt unregistered [12243.662327] Key type ._llcrypt registered [12243.668728] Key type .llcrypt registered [12243.783518] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12244.209611] Key type .llcrypt unregistered [12244.214832] Key type ._llcrypt unregistered [12325.834704] Key type ._llcrypt registered [12325.839790] Key type .llcrypt registered [12326.111101] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12326.550894] Key type .llcrypt unregistered [12326.557446] Key type ._llcrypt unregistered [12363.603572] Key type ._llcrypt registered [12363.618400] Key type .llcrypt registered [12363.840239] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12364.297472] Key type .llcrypt unregistered [12364.302518] Key type ._llcrypt unregistered [12396.837341] Key type ._llcrypt registered [12396.839059] Key type .llcrypt registered [12396.963455] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12397.241512] Key type .llcrypt unregistered [12397.245213] Key type ._llcrypt unregistered [12478.595859] Key type ._llcrypt registered [12478.597925] Key type .llcrypt registered [12478.726617] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12479.084843] Key type .llcrypt unregistered [12479.091732] Key type ._llcrypt unregistered [12511.335446] Key type ._llcrypt registered [12511.338189] Key type .llcrypt registered [12511.503066] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12512.004418] Key type .llcrypt unregistered [12512.010170] Key type ._llcrypt unregistered [12549.209787] Key type ._llcrypt registered [12549.214233] Key type .llcrypt registered [12549.390296] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12549.775353] Key type .llcrypt unregistered [12549.777639] Key type ._llcrypt unregistered [12582.825254] Key type ._llcrypt registered [12582.827219] Key type .llcrypt registered [12582.894811] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12583.261326] Key type .llcrypt unregistered [12583.264349] Key type ._llcrypt unregistered [12614.597801] Key type ._llcrypt registered [12614.599993] Key type .llcrypt registered [12614.721872] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12615.101541] Key type .llcrypt unregistered [12615.103052] Key type ._llcrypt unregistered [12647.428920] Key type ._llcrypt registered [12647.431093] Key type .llcrypt registered [12647.513000] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12647.759143] Key type .llcrypt unregistered [12647.762187] Key type ._llcrypt unregistered [12708.628452] Key type ._llcrypt registered [12708.630252] Key type .llcrypt registered [12708.736355] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12709.024346] Key type .llcrypt unregistered [12709.027532] Key type ._llcrypt unregistered [12737.953685] Key type ._llcrypt registered [12737.956768] Key type .llcrypt registered [12738.126334] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12738.462283] Key type .llcrypt unregistered [12738.473063] Key type ._llcrypt unregistered [12766.169565] Key type ._llcrypt registered [12766.175160] Key type .llcrypt registered [12766.256677] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12766.679630] Key type .llcrypt unregistered [12766.685788] Key type ._llcrypt unregistered [12818.113208] Key type ._llcrypt registered [12818.115450] Key type .llcrypt registered [12818.240767] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [12832.523762] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [12833.684555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12833.728035] alg: No test for adler32 (adler32-zlib) [12834.763526] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [12834.983775] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [12836.671939] Key type lgssc registered [12837.733739] Lustre: Echo OBD driver; http://www.lustre.org/ [12872.564985] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12872.593711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12873.886576] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12873.922886] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12874.009221] Lustre: tst250fs-MDT0000: new disk, initializing [12874.098205] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12874.119310] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12879.162724] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12890.665752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12890.836360] Lustre: 357767:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12890.938517] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12890.942687] Lustre: Skipped 1 previous similar message [12891.014286] Lustre: tst250fs-MDT0001: new disk, initializing [12891.131278] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12891.178885] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12891.193863] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12895.771431] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12900.749803] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12907.354110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12907.697726] Lustre: tst250fs-OST0000: new disk, initializing [12907.701156] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12907.718586] Lustre: 359371:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12907.725465] Lustre: tst250fs-OST0000: Not available for connect from 0@lo (not set up) [12907.806615] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12913.204324] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12913.228817] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12913.263786] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [12913.955639] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12924.774581] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12925.069623] Lustre: tst250fs-OST0001: new disk, initializing [12925.073905] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12925.080451] Lustre: 360222:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12925.176351] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12931.701845] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12935.258820] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12935.267122] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12935.357788] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12937.169821] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12939.587895] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12939.593261] Lustre: Skipped 2 previous similar messages [12961.260062] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12961.273775] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12961.282156] Lustre: Skipped 1 previous similar message [12966.102677] Lustre: server umount tst250fs-MDT0000 complete [12970.350432] LustreError: 359368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245194 with bad export cookie 14374009849608007099 [12970.362122] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12970.368741] LustreError: 359368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12970.605171] Lustre: server umount tst250fs-MDT0001 complete [12984.866893] Lustre: server umount tst250fs-OST0000 complete [12987.809182] Lustre: 355993:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781245195/real 1781245195] req@ffff9e4049886940 x1867770819053184/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781245211 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12987.844351] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12987.856779] Lustre: Skipped 2 previous similar messages [12988.518740] Lustre: server umount tst250fs-OST0001 complete [13005.045621] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [13012.296796] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [13039.362863] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 02:21:03 (1781245263) [13040.720663] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13056.648475] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [13065.953725] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [13111.486567] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12847 sec ========= 02:22:14 (1781245334) [13113.343735] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:22:16 (1781245336) === [13116.948393] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:22:20 (1781245340) === [13160.559463] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [13163.517904] Key type lgssc unregistered [13163.793371] LNet: 369253:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13163.807893] LNetError: 369253:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13163.838195] LNet: Removed LNI 192.168.203.135@tcp [13164.680235] Key type .llcrypt unregistered [13164.682448] Key type ._llcrypt unregistered