[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 501129420 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003252] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008652] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011012] pid_max: default: 32768 minimum: 301 [ 0.012186] LSM: Security Framework initializing [ 0.013098] Yama: becoming mindful. [ 0.014083] SELinux: Initializing. [ 0.015131] *** VALIDATE selinux *** [ 0.024000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029072] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030156] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031200] *** VALIDATE tmpfs *** [ 0.032067] *** VALIDATE proc *** [ 0.034261] *** VALIDATE cgroup *** [ 0.035019] *** VALIDATE cgroup2 *** [ 0.036346] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037207] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039044] Spectre V2 : User space: Vulnerable [ 0.040023] Speculative Store Bypass: Vulnerable [ 0.043614] debug: unmapping init [mem 0xffffffffb2659000-0xffffffffb2660fff] [ 0.046175] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047823] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048030] ... version: 2 [ 0.049018] ... bit width: 48 [ 0.050029] ... generic registers: 4 [ 0.051019] ... value mask: 0000ffffffffffff [ 0.052024] ... max period: 00007fffffffffff [ 0.053026] ... fixed-purpose events: 3 [ 0.054021] ... event mask: 000000070000000f [ 0.055407] rcu: Hierarchical SRCU implementation. [ 0.057864] smp: Bringing up secondary CPUs ... [ 0.058756] x86: Booting SMP configuration: [ 0.059041] .... node #0, CPUs: #1 #2 #3 [ 0.063487] smp: Brought up 1 node, 4 CPUs [ 0.065021] smpboot: Max logical packages: 1 [ 0.066016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.166541] node 0 deferred pages initialised in 96ms [ 0.170149] devtmpfs: initialized [ 0.172363] x86/mm: Memory block size: 128MB [ 0.176205] gcov: version magic: 0x41383552 [ 0.178358] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.182150] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.185395] pinctrl core: initialized pinctrl subsystem [ 0.187229] [ 0.187847] ************************************************************* [ 0.190022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.192017] ** ** [ 0.195018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.197017] ** ** [ 0.199017] ** This means that this kernel is built to expose internal ** [ 0.201019] ** IOMMU data structures, which may compromise security on ** [ 0.204020] ** your system. ** [ 0.206016] ** ** [ 0.209019] ** If you see this message and you are not debugging the ** [ 0.211017] ** kernel, report this immediately to your vendor! ** [ 0.214018] ** ** [ 0.215012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.217012] ************************************************************* [ 0.218776] NET: Registered protocol family 16 [ 0.221502] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.224083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.226087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.230150] cpuidle: using governor menu [ 0.231611] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.232905] PCI: Using configuration type 1 for base access [ 0.233146] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.241071] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.243050] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.247040] cryptd: max_cpu_qlen set to 1000 [ 0.249381] ACPI: Added _OSI(Module Device) [ 0.251022] ACPI: Added _OSI(Processor Device) [ 0.253017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.255019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.259627] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.267139] ACPI: Interpreter enabled [ 0.268066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.270020] ACPI: Using IOAPIC for interrupt routing [ 0.272192] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.275425] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.285080] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.287068] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.290031] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.294116] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.298370] acpiphp: Slot [2] registered [ 0.300194] acpiphp: Slot [3] registered [ 0.302402] acpiphp: Slot [4] registered [ 0.304122] acpiphp: Slot [5] registered [ 0.305202] acpiphp: Slot [6] registered [ 0.307204] acpiphp: Slot [7] registered [ 0.309180] acpiphp: Slot [8] registered [ 0.310222] acpiphp: Slot [9] registered [ 0.313215] acpiphp: Slot [10] registered [ 0.314103] acpiphp: Slot [11] registered [ 0.316122] acpiphp: Slot [12] registered [ 0.317110] acpiphp: Slot [13] registered [ 0.319150] acpiphp: Slot [14] registered [ 0.320166] acpiphp: Slot [15] registered [ 0.322176] acpiphp: Slot [16] registered [ 0.324137] acpiphp: Slot [17] registered [ 0.325143] acpiphp: Slot [18] registered [ 0.327170] acpiphp: Slot [19] registered [ 0.329164] acpiphp: Slot [20] registered [ 0.330130] acpiphp: Slot [21] registered [ 0.331096] acpiphp: Slot [22] registered [ 0.332094] acpiphp: Slot [23] registered [ 0.333117] acpiphp: Slot [24] registered [ 0.335117] acpiphp: Slot [25] registered [ 0.336154] acpiphp: Slot [26] registered [ 0.338135] acpiphp: Slot [27] registered [ 0.340173] acpiphp: Slot [28] registered [ 0.341124] acpiphp: Slot [29] registered [ 0.343140] acpiphp: Slot [30] registered [ 0.344123] acpiphp: Slot [31] registered [ 0.345104] PCI host bridge to bus 0000:00 [ 0.346023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.349046] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.351030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.353036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.355031] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.358035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.360246] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.366803] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.371866] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.381763] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.387516] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.389027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.391027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.394032] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.397113] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.399809] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.402048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.404747] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.409013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.419848] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.426024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.432929] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.441024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.452023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.470024] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.479000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.486028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.492021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.510037] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.523943] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.531020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.537021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.552030] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.565064] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.571018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.578023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.593025] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.603808] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.611018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.619020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.629016] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.640000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.645021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.653019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.666021] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.679207] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.681438] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.684399] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.686416] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.689188] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.693196] iommu: Default domain type: Passthrough [ 0.694686] SCSI subsystem initialized [ 0.696304] ACPI: bus type USB registered [ 0.698224] usbcore: registered new interface driver usbfs [ 0.701154] usbcore: registered new interface driver hub [ 0.703167] usbcore: registered new device driver usb [ 0.705245] pps_core: LinuxPPS API ver. 1 registered [ 0.707318] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.711100] PTP clock support registered [ 0.713078] EDAC MC: Ver: 3.0.0 [ 0.715179] PCI: Using ACPI for IRQ routing [ 0.716764] NetLabel: Initializing [ 0.717000] NetLabel: domain hash size = 128 [ 0.717021] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.720126] NetLabel: unlabeled traffic allowed by default [ 0.723402] vgaarb: loaded [ 0.725227] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.727023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.737120] clocksource: Switched to clocksource kvm-clock [ 0.847974] VFS: Disk quotas dquot_6.6.0 [ 0.849399] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.852127] *** VALIDATE ramfs *** [ 0.853507] *** VALIDATE hugetlbfs *** [ 0.855444] pnp: PnP ACPI init [ 0.858171] pnp: PnP ACPI: found 6 devices [ 0.879993] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.883432] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.885578] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.887959] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.890439] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.892945] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.895471] NET: Registered protocol family 2 [ 0.898332] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.904045] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.908684] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.914669] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.918345] TCP: Hash tables configured (established 65536 bind 65536) [ 0.922417] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.926686] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.929839] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.933079] NET: Registered protocol family 1 [ 0.935903] RPC: Registered named UNIX socket transport module. [ 0.938033] RPC: Registered udp transport module. [ 0.939620] RPC: Registered tcp transport module. [ 0.941304] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.943978] NET: Registered protocol family 44 [ 0.945921] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.948180] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.950517] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.953064] PCI: CLS 0 bytes, default 64 [ 0.955520] Unpacking initramfs... [ 2.398492] debug: unmapping init [mem 0xffff8c727cc54000-0xffff8c727ffbffff] [ 2.403038] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.405482] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.408288] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.956953] Initialise system trusted keyrings [ 2.958852] Key type blacklist registered [ 2.961029] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.970116] zbud: loaded [ 2.973624] *** VALIDATE nfs *** [ 2.975070] *** VALIDATE nfs4 *** [ 2.977625] pstore: using deflate compression [ 2.980888] Platform Keyring initialized [ 3.096809] NET: Registered protocol family 38 [ 3.098529] Key type asymmetric registered [ 3.100190] Asymmetric key parser 'x509' registered [ 3.102102] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.106527] io scheduler mq-deadline registered [ 3.108468] io scheduler kyber registered [ 3.110291] io scheduler bfq registered [ 3.112619] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.116048] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.118225] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.121045] ACPI: Power Button [PWRF] [ 3.214390] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.307435] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.490995] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.588208] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.807664] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.839961] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.870247] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.876305] Non-volatile memory driver v1.3 [ 3.878659] Linux agpgart interface v0.103 [ 3.910985] virtio_blk virtio1: [vda] 133832 512-byte logical blocks (68.5 MB/65.3 MiB) [ 3.913757] vda: detected capacity change from 0 to 68521984 [ 3.931563] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.935173] vdb: detected capacity change from 0 to 1073741824 [ 3.952614] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.955608] vdc: detected capacity change from 0 to 2621440000 [ 3.973206] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.977229] vdd: detected capacity change from 0 to 2621440000 [ 3.998827] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.001958] vde: detected capacity change from 0 to 4294967296 [ 4.023086] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.027922] vdf: detected capacity change from 0 to 4294967296 [ 4.034670] libphy: Fixed MDIO Bus: probed [ 4.040163] usbcore: registered new interface driver usbserial_generic [ 4.044099] usbserial: USB Serial support registered for generic [ 4.047023] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.053293] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.055484] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.058821] mousedev: PS/2 mouse device common for all mice [ 4.062842] rtc_cmos 00:05: RTC can wake from S4 [ 4.067704] rtc_cmos 00:05: registered as rtc0 [ 4.069810] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.075826] intel_pstate: CPU model not supported [ 4.080899] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.083528] hid: raw HID events driver (C) Jiri Kosina [ 4.089735] usbcore: registered new interface driver usbhid [ 4.092585] usbhid: USB HID core driver [ 4.094476] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.095109] drop_monitor: Initializing network drop monitor service [ 4.099926] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.101544] Initializing XFRM netlink socket [ 4.107782] NET: Registered protocol family 10 [ 4.110862] Segment Routing with IPv6 [ 4.113264] NET: Registered protocol family 17 [ 4.141963] mpls_gso: MPLS GSO support [ 4.153152] RAS: Correctable Errors collector initialized. [ 4.155436] AVX version of gcm_enc/dec engaged. [ 4.157226] AES CTR mode by8 optimization enabled [ 4.232726] sched_clock: Marking stable (4232701614, 0)->(5179016457, -946314843) [ 4.236474] registered taskstats version 1 [ 4.238658] Loading compiled-in X.509 certificates [ 4.241041] zswap: loaded using pool lzo/zbud [ 4.269894] Key type big_key registered [ 4.281686] Key type encrypted registered [ 4.283351] ima: No TPM chip found, activating TPM-bypass! [ 4.285553] ima: Allocated hash algorithm: sha1 [ 4.287349] ima: No architecture policies found [ 4.289213] evm: Initialising EVM extended attributes: [ 4.291250] evm: security.selinux [ 4.292562] evm: security.ima [ 4.293736] evm: security.capability [ 4.295130] evm: HMAC attrs: 0x1 [ 4.298830] rtc_cmos 00:05: setting system clock to 2025-11-16 23:29:12 UTC (1763335752) [ 4.305287] debug: unmapping init [mem 0xffffffffb3603000-0xffffffffb37fffff] [ 4.308586] debug: unmapping init [mem 0xffffffffb2382000-0xffffffffb2658fff] [ 4.318391] Write protecting the kernel read-only data: 28672k [ 4.321690] debug: unmapping init [mem 0xffffffffb0a03000-0xffffffffb0bfffff] [ 4.324587] debug: unmapping init [mem 0xffffffffb1314000-0xffffffffb13fffff] [ 4.366181] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.373690] systemd[1]: Detected virtualization kvm. [ 4.375666] systemd[1]: Detected architecture x86-64. [ 4.377560] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.402304] systemd[1]: No hostname configured. [ 4.403932] systemd[1]: Set hostname to . [ 4.406425] random: systemd: uninitialized urandom read (16 bytes read) [ 4.408073] systemd[1]: Initializing machine ID from random generator. [ 4.463237] random: ln: uninitialized urandom read (6 bytes read) [ 4.558784] random: systemd: uninitialized urandom read (16 bytes read) [ 4.561415] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.567243] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.571245] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.227801] device-mapper: uevent: version 1.0.3 [ 5.230358] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.916281] virtio_net virtio0 ens2: renamed from eth0 [ 5.939459] scsi host0: ata_piix [ 5.945689] scsi host1: ata_piix [ 5.947888] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.950243] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 5.983374] random: fast init done [ 10.322091] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.803156] random: crng init done [ 10.804525] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.256797] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.585424] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.902660] SELinux: Disabled at runtime. [ 12.969623] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.982685] systemd[1]: Detected virtualization kvm. [ 12.984970] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.639087] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.642976] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.652417] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.657984] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.661939] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.673770] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.698536] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 13.859773] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 14.165099] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 14.584592] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.603306] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.812299] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.831321] EDAC sbridge: Ver: 1.1.2 [ 17.161645] Key type dns_resolver registered [ 17.465741] NFS: Registering the id_resolver key type [ 17.467925] Key type id_resolver registered [ 17.469553] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg255-server login: [ 30.244239] hrtimer: interrupt took 4412842 ns [ 75.083178] libcfs: loading out-of-tree module taints kernel. [ 75.121940] Key type ._llcrypt registered [ 75.123496] Key type .llcrypt registered [ 75.214796] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 88.713567] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 89.633789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 89.650692] alg: No test for adler32 (adler32-zlib) [ 90.886543] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 91.578392] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 93.255193] Key type lgssc registered [ 94.302213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.629469] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 106.249563] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 112.202890] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 117.756735] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 123.617618] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 135.165590] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 143.634410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 143.680168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 143.692712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 144.857084] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 144.888570] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 144.961119] Lustre: lustre-MDT0000: new disk, initializing [ 145.024874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 145.038737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 147.777420] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 156.977938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 157.026535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 157.076802] Lustre: 6508:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 157.100463] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 157.103500] Lustre: Skipped 1 previous similar message [ 157.178539] Lustre: lustre-MDT0001: new disk, initializing [ 157.222922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 157.252712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 157.261284] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 159.921759] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 163.635194] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 169.947804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 170.025365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 170.191648] Lustre: lustre-OST0000: new disk, initializing [ 170.198270] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 170.253634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 173.759053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 173.763718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 173.820512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 173.991525] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.229242] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 184.297229] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 184.407890] Lustre: lustre-OST0001: new disk, initializing [ 184.412369] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 184.472040] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 188.341615] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.531186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 192.541738] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 192.581300] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 197.291767] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 209.363757] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 218.726836] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing check_logdir /tmp/testlogs/ [ 221.779460] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing yml_node [ 225.494991] Lustre: DEBUG MARKER: Client: 2.16.61.52 [ 227.386229] Lustre: DEBUG MARKER: MDS: 2.16.61.52 [ 228.990186] Lustre: DEBUG MARKER: OSS: 2.16.61.52 [ 230.014820] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 18:32:58 EST 2025 [ 241.727529] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 242.632643] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 259.046764] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 259.048269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 259.059571] Lustre: Skipped 3 previous similar messages [ 259.071331] Lustre: Skipped 2 previous similar messages [ 264.161151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 264.165329] Lustre: Skipped 3 previous similar messages [ 269.283950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 270.303431] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 270.386807] LustreError: 12215:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 270.447620] Lustre: server umount lustre-MDT0000 complete [ 272.965223] LustreError: 6502:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336021 with bad export cookie 10724813882044355341 [ 272.968293] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 272.970530] LustreError: 6502:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 274.400935] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 274.404023] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 274.406582] Lustre: Skipped 1 previous similar message [ 274.415497] Lustre: Skipped 3 previous similar messages [ 279.130750] LustreError: 12419:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 279.140729] LustreError: 12419:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 279.265993] Lustre: server umount lustre-MDT0001 complete [ 282.072071] LustreError: 12620:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 282.085104] LustreError: 12620:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 282.131583] Lustre: server umount lustre-OST0000 complete [ 284.882840] LustreError: 12820:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 284.887884] LustreError: 12820:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 284.993679] Lustre: server umount lustre-OST0001 complete [ 289.479489] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 295.675677] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 302.325336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 307.720656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 312.327926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 317.465718] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 323.201679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 323.263941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 323.492960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 323.533242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 323.616648] Lustre: lustre-MDT0000: new disk, initializing [ 323.748210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 323.765511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 327.092914] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 336.023354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 336.092820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 336.166984] Lustre: 15004:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 336.231711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 336.234859] Lustre: Skipped 1 previous similar message [ 336.304796] Lustre: lustre-MDT0001: new disk, initializing [ 336.410175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 336.432915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 336.440881] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 339.543743] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 345.312522] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 348.518211] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 353.456434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 353.529628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 353.782785] Lustre: lustre-OST0000: new disk, initializing [ 353.796794] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 353.873849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 355.508146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 355.513175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 355.622862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 358.081389] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 364.672602] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 367.868784] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 368.033175] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 370.739542] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 370.879494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 376.289670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 376.290344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 376.301201] LustreError: Skipped 1 previous similar message [ 376.302286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 376.307921] Lustre: Skipped 1 previous similar message [ 376.315444] Lustre: Skipped 4 previous similar messages [ 378.303238] LustreError: 17946:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 378.309460] LustreError: 17946:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 378.458499] Lustre: server umount lustre-OST0000 complete [ 382.434955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 382.439637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.539345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.539412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 386.548222] Lustre: Skipped 1 previous similar message [ 387.832986] LustreError: 18150:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 387.837867] LustreError: 18150:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 387.920701] Lustre: server umount lustre-MDT0000 complete [ 390.959812] LustreError: 16754:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336139 with bad export cookie 10724813882044356874 [ 390.961884] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 390.966193] LustreError: 16754:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 391.246679] Lustre: server umount lustre-MDT0001 complete [ 396.487838] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 18:35:44 (1763336144) [ 411.979890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.353151] LustreError: 19487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 412.371297] LustreError: 19487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 412.445316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.733638] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 417.762434] LustreError: 19487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 421.857718] LustreError: 19487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 422.972235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 426.945220] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 431.784041] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 434.604898] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 436.317541] Lustre: Failing over lustre-MDT0000 [ 436.329657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 436.345660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 436.359661] Lustre: Skipped 1 previous similar message [ 436.446171] LustreError: 20830:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 436.448576] LustreError: 20830:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 436.543461] Lustre: server umount lustre-MDT0000 complete [ 441.826115] LustreError: 20201:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 441.837795] LustreError: 20201:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 446.948442] LustreError: 19487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 446.955241] LustreError: 19487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 455.048173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 455.158546] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 458.610810] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 460.776266] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 460.779624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 460.815908] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 465.249296] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 471.018484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 471.021781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 471.026904] Lustre: Skipped 1 previous similar message [ 471.031754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 483.295185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 483.386950] LustreError: 21810:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 483.441132] LustreError: 21810:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 483.568025] Lustre: server umount lustre-MDT0000 complete [ 486.751874] LustreError: 20831:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336234 with bad export cookie 10724813882044358288 [ 486.755151] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 486.759105] LustreError: 20831:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 487.134221] Lustre: server umount lustre-MDT0001 complete [ 495.165631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.717812] LustreError: 22489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.747264] LustreError: 22489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 495.818335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 495.830966] Lustre: Skipped 2 previous similar messages [ 499.354191] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 506.899909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 510.140585] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 515.031970] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 518.028063] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 519.863677] Lustre: Failing over lustre-MDT0000 [ 520.100918] Lustre: server umount lustre-MDT0000 complete [ 522.721235] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.727929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 522.739961] Lustre: Skipped 1 previous similar message [ 522.753846] LustreError: 22484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.769180] LustreError: 22484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 538.079255] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763336270/real 1763336270] req@ffff8c71c702d500 x1848991640883200/t0(0) o400->MGC192.168.202.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763336286 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 538.102721] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 538.382924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.324219] LustreError: 3650:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8c72f902e680 x1848991640885632/t0(0) o250->MGC192.168.202.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 551.602422] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 552.676377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 552.677494] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 552.688105] Lustre: Skipped 1 previous similar message [ 552.714579] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 557.562940] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 563.169696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 563.171860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 563.192572] Lustre: Skipped 7 previous similar messages [ 566.800882] LustreError: 24814:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 566.804086] LustreError: 24814:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 566.884878] Lustre: server umount lustre-MDT0000 complete [ 568.293257] LustreError: 22483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.307036] LustreError: 22483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 569.840046] LustreError: 23833:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336318 with bad export cookie 10724813882044359219 [ 569.842818] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 569.856214] LustreError: 23833:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 570.094829] Lustre: server umount lustre-MDT0001 complete [ 576.920720] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 18:38:45 (1763336325) [ 590.279838] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 598.292588] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 605.426952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 610.425657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 615.168278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 620.739676] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 628.149988] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 636.878215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 636.925108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 637.090265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 637.112602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 637.156856] Lustre: lustre-MDT0000: new disk, initializing [ 637.233095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 637.238593] Lustre: Skipped 2 previous similar messages [ 637.261052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 640.297578] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 648.900799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 648.979853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 649.050597] Lustre: 28483:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 649.319089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 652.588396] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 659.418738] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 662.303383] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 667.894456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 667.965880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 669.321854] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 669.376117] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 669.432109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 669.447151] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 672.927954] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 678.999171] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 683.815638] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 700.385214] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 700.387852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 700.394944] LustreError: Skipped 1 previous similar message [ 700.400091] Lustre: Skipped 3 previous similar messages [ 700.404589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 700.407224] Lustre: Skipped 1 previous similar message [ 706.030145] LustreError: 31140:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 706.033750] LustreError: 31140:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 706.169781] Lustre: server umount lustre-OST0000 complete [ 716.304627] Lustre: server umount lustre-MDT0000 complete [ 719.392899] LustreError: 28475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336467 with bad export cookie 10724813882044359618 [ 719.398806] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 719.398889] LustreError: 28475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 719.668507] Lustre: server umount lustre-MDT0001 complete [ 726.832746] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 729.289214] Key type lgssc unregistered [ 729.547719] LNet: 32076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.551271] LNetError: 32076:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 729.565972] LNet: Removed LNI 192.168.202.155@tcp [ 730.222140] Key type .llcrypt unregistered [ 730.225105] Key type ._llcrypt unregistered [ 747.641556] Key type ._llcrypt registered [ 747.643282] Key type .llcrypt registered [ 747.754696] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 748.418941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 748.446252] alg: No test for adler32 (adler32-zlib) [ 749.450253] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 749.655210] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 751.311190] Key type lgssc registered [ 752.190259] Lustre: Echo OBD driver; http://www.lustre.org/ [ 758.687307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 758.707642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 760.127555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 763.405153] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 770.297622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 770.607220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 773.404944] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 776.658689] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 779.339627] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 784.418877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 784.722313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 789.224780] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 793.845962] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 794.999900] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 804.752980] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 18:42:33 (1763336553) [ 806.051185] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 807.322905] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 812.700400] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 18:42:40 (1763336560) [ 819.232549] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 18:42:47 (1763336567) [ 819.976652] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 827.074181] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 827.080108] Lustre: Skipped 2 previous similar messages [ 828.820386] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 18:42:56 (1763336576) [ 832.352911] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 832.355624] Lustre: Skipped 3 previous similar messages [ 847.503263] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 847.507367] Lustre: Skipped 6 previous similar messages [ 849.875896] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 18:43:18 (1763336598) [ 852.364422] Lustre: *** cfs_fail_loc=131b, val=2*** [ 855.997024] Lustre: *** cfs_fail_loc=131b, val=2*** [ 864.111764] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 18:43:32 (1763336612) [ 866.000314] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 866.003023] Lustre: Skipped 3 previous similar messages [ 877.524954] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 18:43:45 (1763336625) [ 892.386250] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 892.392405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 892.398982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 892.405245] Lustre: Skipped 1 previous similar message [ 892.414048] Lustre: Skipped 1 previous similar message [ 896.072096] LustreError: 40006:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 896.233421] Lustre: server umount lustre-OST0000 complete [ 902.624605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 902.630082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 902.643270] Lustre: Skipped 1 previous similar message [ 906.432577] LustreError: 40207:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 906.435520] LustreError: 40207:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 906.509186] Lustre: server umount lustre-MDT0000 complete [ 907.746773] LustreError: 35064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 907.762558] LustreError: 35064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 909.979252] LustreError: 33427:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336658 with bad export cookie 3481806252347738103 [ 909.985439] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 909.988140] LustreError: 33427:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 910.126538] LustreError: 40411:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 910.128926] LustreError: 40411:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 910.272977] Lustre: server umount lustre-MDT0001 complete [ 918.740108] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 921.056611] Key type lgssc unregistered [ 921.367987] LNet: 40942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.376771] LNetError: 40942:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.404841] LNet: Removed LNI 192.168.202.155@tcp [ 923.062225] Key type .llcrypt unregistered [ 923.068091] Key type ._llcrypt unregistered [ 936.204158] Key type ._llcrypt registered [ 936.205892] Key type .llcrypt registered [ 936.305103] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 948.237674] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 948.855373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 948.862140] alg: No test for adler32 (adler32-zlib) [ 949.931319] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 950.249501] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 951.943226] Key type lgssc registered [ 953.172585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 962.113774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 967.946293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 972.550459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 978.135042] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 983.987931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 984.044044] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 984.059277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 985.262147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 985.283166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 985.363554] Lustre: lustre-MDT0000: new disk, initializing [ 985.447894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 985.471995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 988.360657] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 996.225322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 996.260717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 996.410962] Lustre: 44050:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 996.486156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 996.489156] Lustre: Skipped 1 previous similar message [ 996.632465] Lustre: lustre-MDT0001: new disk, initializing [ 996.734401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 996.807756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 996.814429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1000.648283] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1007.693591] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1011.628730] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1018.653863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1018.727666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1019.107289] Lustre: lustre-OST0000: new disk, initializing [ 1019.138891] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1019.294641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1024.846213] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1027.158213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1027.178828] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1027.240089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1031.386316] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1034.910178] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1035.075855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1038.134837] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1038.328345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1042.921043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1042.922625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1042.926565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1042.929989] Lustre: Skipped 1 previous similar message [ 1042.943731] LustreError: Skipped 1 previous similar message [ 1048.038053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1048.041148] Lustre: Skipped 2 previous similar messages [ 1053.154837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1053.163541] Lustre: Skipped 1 previous similar message [ 1054.175168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1054.217226] LustreError: 46986:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1054.330912] Lustre: server umount lustre-OST0000 complete [ 1058.273757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1058.273877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1058.277680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1058.295970] Lustre: Skipped 1 previous similar message [ 1063.392785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1063.395957] Lustre: Skipped 2 previous similar messages [ 1071.071315] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1071.174432] LustreError: 47189:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1071.178665] LustreError: 47189:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1071.265394] Lustre: server umount lustre-MDT0000 complete [ 1074.530822] LustreError: 44040:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336822 with bad export cookie 390786968142486392 [ 1074.536677] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1074.541836] LustreError: 44040:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1074.653286] LustreError: 47392:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1074.660298] LustreError: 47392:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1074.817408] Lustre: server umount lustre-MDT0001 complete [ 1082.159807] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 18:47:10 (1763336830) [ 1087.962958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1088.309997] LustreError: 48112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1088.314891] LustreError: 48112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1088.384209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1091.498423] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1093.603079] LustreError: 48111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1097.852340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1098.089595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1100.936698] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1104.947885] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1107.810842] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1112.856707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1113.183163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1117.566729] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1121.687977] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1128.858476] LustreError: 50306:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1129.526664] Lustre: Modifying parameter general.timeout=1 in log params [ 1130.908625] Lustre: Modifying parameter general.timeout=3 in log params [ 1130.910993] Lustre: Skipped 1 previous similar message [ 1133.577633] Lustre: Modifying parameter general.timeout=7 in log params [ 1133.580391] Lustre: Skipped 3 previous similar messages [ 1137.986285] Lustre: Modifying parameter general.timeout=14 in log params [ 1137.988954] Lustre: Skipped 6 previous similar messages [ 1146.593915] Lustre: Modifying parameter general.timeout=27 in log params [ 1146.596338] Lustre: Skipped 12 previous similar messages [ 1162.678807] Lustre: Modifying parameter general.timeout=44 in log params [ 1162.681484] Lustre: Skipped 16 previous similar messages [ 1195.040944] Lustre: Modifying parameter general.timeout=88 in log params [ 1195.043720] Lustre: Skipped 43 previous similar messages [ 1235.050748] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 18:49:43 (1763336983) [ 1237.420036] Lustre: 57241:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1239.110614] Lustre: 57339:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1242.519135] Lustre: 57549:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1242.543220] Lustre: 57549:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 1 previous similar message [ 1259.089123] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 in log params [ 1259.099872] Lustre: Skipped 60 previous similar messages [ 1266.050464] Lustre: 59091:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1266.056182] Lustre: 59091:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 2 previous similar messages [ 1266.074803] LustreError: 59091:0:(mgs_handler.c:1126:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1269.057397] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 18:50:17 (1763337017) [ 1277.407965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1277.412106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1277.421345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1277.424743] Lustre: Skipped 2 previous similar messages [ 1277.441965] LustreError: 59794:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1277.445484] LustreError: 59794:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1277.510911] Lustre: server umount lustre-MDT0000 complete [ 1278.946676] LustreError: 48111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.959475] LustreError: 48111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1279.841038] LustreError: 48091:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337028 with bad export cookie 390786968142487274 [ 1279.848973] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1279.849245] LustreError: 48091:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1284.064459] LustreError: 48108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1284.068780] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.091131] Lustre: Skipped 2 previous similar messages [ 1286.043436] LustreError: 59994:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1286.046398] LustreError: 59994:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1286.185932] Lustre: server umount lustre-MDT0001 complete [ 1289.017343] Lustre: server umount lustre-OST0000 complete [ 1295.157308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1298.283147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1300.648885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1303.081274] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1308.368091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1308.682513] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1308.942733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1312.438727] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1319.388240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1319.481318] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1319.518332] Lustre: 61645:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1322.750662] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1327.076067] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1329.905460] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1335.115761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1335.238091] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1340.219577] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1345.575446] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1347.020425] Lustre: 63874:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1352.677766] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1352.681502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1352.699918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1352.706257] Lustre: Skipped 1 previous similar message [ 1357.281359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1358.332076] LustreError: 64131:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1358.338219] LustreError: 64131:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1358.436177] Lustre: server umount lustre-OST0000 complete [ 1362.915622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.936758] Lustre: Skipped 1 previous similar message [ 1376.223600] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1376.330940] LustreError: 64333:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1376.341337] LustreError: 64333:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1376.438685] Lustre: server umount lustre-MDT0000 complete [ 1378.275537] LustreError: 61649:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1378.287047] LustreError: 61649:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1379.578850] LustreError: 61636:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337127 with bad export cookie 390786968142490137 [ 1379.579160] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1379.592904] LustreError: 61636:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1379.965675] Lustre: server umount lustre-MDT0001 complete [ 1387.764075] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 1389.825629] Key type lgssc unregistered [ 1390.056613] LNet: 65067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1390.065912] LNetError: 65067:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1390.078742] LNet: Removed LNI 192.168.202.155@tcp [ 1390.795329] Key type .llcrypt unregistered [ 1390.801333] Key type ._llcrypt unregistered [ 1408.477266] Key type ._llcrypt registered [ 1408.479547] Key type .llcrypt registered [ 1408.564868] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 1409.151883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1409.266172] alg: No test for adler32 (adler32-zlib) [ 1410.296246] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 1410.509776] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 1412.167201] Key type lgssc registered [ 1412.972380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1417.826305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1417.850377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1419.188215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.345907] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1427.186463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1427.562679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1430.115719] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1434.168081] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1436.283054] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1441.295531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1441.713470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1446.874269] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1451.513427] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1459.272220] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1467.363214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1467.372351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1472.483395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1472.483394] LustreError: 69147:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1472.490689] Lustre: Skipped 4 previous similar messages [ 1472.552490] Lustre: server umount lustre-MDT0000 complete [ 1476.116901] LustreError: 68791:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337224 with bad export cookie 11159716309365006463 [ 1476.125039] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1476.127186] LustreError: 68791:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1476.276785] LustreError: 69351:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1476.289764] LustreError: 69351:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1476.461154] Lustre: server umount lustre-MDT0001 complete [ 1488.848723] LustreError: 69551:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1488.851880] LustreError: 69551:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1488.978114] Lustre: server umount lustre-OST0000 complete [ 1496.137302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1499.395488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1502.440767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1505.534488] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1510.915871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1511.138533] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1511.169989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1511.363315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1514.406199] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1521.370442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1521.440516] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1521.459831] Lustre: 71003:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1521.938122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1525.312318] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1529.887940] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1532.887085] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1538.637131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1538.827796] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1539.254546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1544.230846] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1548.828867] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1549.923265] Lustre: 73247:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1551.293594] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1556.346171] LustreError: 73495:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1556.358844] LustreError: 73495:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1556.460083] Lustre: server umount lustre-OST0000 complete [ 1562.594099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1562.600456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1562.620316] Lustre: Skipped 2 previous similar messages [ 1562.640267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1564.137083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1569.249727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1569.254153] Lustre: Skipped 3 previous similar messages [ 1570.286842] LustreError: 73702:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1570.294397] LustreError: 73702:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1570.345857] Lustre: server umount lustre-MDT0000 complete [ 1573.093899] LustreError: 70997:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337321 with bad export cookie 11159716309365007296 [ 1573.097526] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1573.105680] LustreError: 70997:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1583.594163] LustreError: 73904:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1583.613515] LustreError: 73904:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1583.799281] Lustre: server umount lustre-MDT0001 complete [ 1592.431912] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 1594.600969] Key type lgssc unregistered [ 1594.846563] LNet: 74437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1594.861297] LNetError: 74437:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1594.878672] LNet: Removed LNI 192.168.202.155@tcp [ 1595.587405] Key type .llcrypt unregistered [ 1595.589096] Key type ._llcrypt unregistered [ 1616.554425] Key type ._llcrypt registered [ 1616.556484] Key type .llcrypt registered [ 1616.642631] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 1617.853380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1617.859218] alg: No test for adler32 (adler32-zlib) [ 1618.945351] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 1619.184609] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 1620.897117] Key type lgssc registered [ 1622.248877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1628.745452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1628.769950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1630.168532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1633.092896] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1639.909770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1640.398131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1643.602617] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1648.732133] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1652.160728] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1657.852797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1658.256605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1661.998218] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1666.344288] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1681.394435] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1682.254525] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1683.682969] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1683.685409] Lustre: Skipped 1 previous similar message [ 1686.000264] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1686.009567] Lustre: Skipped 2 previous similar messages [ 1690.064265] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1690.069820] Lustre: Skipped 4 previous similar messages [ 1698.675713] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1698.678641] Lustre: Skipped 11 previous similar messages [ 1715.178255] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1715.182389] Lustre: Skipped 18 previous similar messages [ 1747.804892] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1747.809618] Lustre: Skipped 38 previous similar messages [ 1771.049234] Lustre: 83621:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1773.471339] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 18:58:41 (1763337521) [ 1774.759083] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1776.732765] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 18:58:44 (1763337524) [ 1783.269300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1783.272943] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1783.306419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1784.292248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1784.292499] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1784.294419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1785.984849] LustreError: 83917:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1786.139772] Lustre: server umount lustre-OST0000 complete [ 1794.016696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1794.034705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1794.048191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1794.051320] Lustre: Skipped 1 previous similar message [ 1795.653939] LustreError: 84118:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1795.673924] LustreError: 84118:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1795.860720] Lustre: server umount lustre-MDT0000 complete [ 1799.398651] LustreError: 75815:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337547 with bad export cookie 7063528007590919975 [ 1799.407205] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1799.414155] LustreError: 75815:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1799.538286] LustreError: 84321:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1799.582420] LustreError: 84321:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1799.705597] Lustre: server umount lustre-MDT0001 complete [ 1809.857637] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 1812.601864] Key type lgssc unregistered [ 1812.870924] LNet: 84852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1812.888305] LNetError: 84852:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1812.906838] LNet: Removed LNI 192.168.202.155@tcp [ 1813.740736] Key type .llcrypt unregistered [ 1813.742533] Key type ._llcrypt unregistered [ 1817.433734] Key type ._llcrypt registered [ 1817.435418] Key type .llcrypt registered [ 1817.532561] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1821.654977] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules [ 1822.359033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1822.400032] alg: No test for adler32 (adler32-zlib) [ 1823.557475] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 1823.847225] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 1825.513064] Key type lgssc registered [ 1826.612290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1827.804638] LustreError: 86018:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 1844.496378] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 1849.977178] LustreError: 86018:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1853.379566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1853.427293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1855.166975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1858.890589] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1865.698775] LustreError: 87025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1865.712155] LustreError: 87025:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1866.535958] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 19:00:14 (1763337614) [ 1870.822108] LustreError: 87025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1871.053719] LustreError: 87057:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1871.067508] LustreError: 87057:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1871.077421] LustreError: 87057:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 1871.193727] LustreError: 87831:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1871.407880] Lustre: server umount lustre-MDT0000 complete [ 1882.591773] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 1884.867771] Key type lgssc unregistered [ 1885.103989] LNet: 88518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1885.114133] LNetError: 88518:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1885.135314] LNet: Removed LNI 192.168.202.155@tcp [ 1885.791290] Key type .llcrypt unregistered [ 1885.801089] Key type ._llcrypt unregistered [ 1899.455590] Key type ._llcrypt registered [ 1899.457304] Key type .llcrypt registered [ 1899.556862] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 1900.430985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1900.522298] alg: No test for adler32 (adler32-zlib) [ 1901.579310] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 1901.783412] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 1903.431173] Key type lgssc registered [ 1904.614532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1912.105752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1912.125888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1913.848215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1917.543037] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1923.052931] LustreError: 89696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1923.091939] LustreError: 89696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1926.120229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1927.057844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1931.026861] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1936.118533] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1940.146175] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1947.965382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1948.680942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1954.359146] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 1959.734474] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2001.468377] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 19:02:29 (1763337749) [ 2003.082080] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2004.890698] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 19:02:33 (1763337753) [ 2008.544462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2008.561994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2008.591970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2014.180073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2014.182711] Lustre: Skipped 3 previous similar messages [ 2014.379872] LustreError: 92618:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2014.468160] Lustre: server umount lustre-MDT0000 complete [ 2017.232485] LustreError: 89677:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337765 with bad export cookie 10259591072810859061 [ 2017.239116] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2017.247902] LustreError: 89677:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2017.361938] LustreError: 92820:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2017.369655] LustreError: 92820:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2017.524063] Lustre: server umount lustre-MDT0001 complete [ 2031.315928] LustreError: 93022:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2031.322200] LustreError: 93022:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2031.699781] Lustre: server umount lustre-OST0000 complete [ 2041.326202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2041.795497] LustreError: 93650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2041.957188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2045.460373] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2047.459646] LustreError: 93650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2052.497149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2052.577633] LustreError: 93650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2053.109321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2056.380615] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2060.590636] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2063.699655] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2068.725824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2074.125116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2074.184570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2074.320341] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2074.327048] LustreError: 93640:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2074.352443] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2074.360659] LustreError: 95513:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 2074.370049] LustreError: 95513:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2074.377124] LustreError: 95513:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2074.433923] Lustre: server umount lustre-OST0000 complete [ 2074.438056] LustreError: 95513:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2081.772938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2081.895705] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2081.899289] LustreError: 93639:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2081.907595] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2081.915321] LustreError: 95977:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 2081.919705] LustreError: 95977:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2081.923695] LustreError: 95977:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2081.959941] Lustre: server umount lustre-OST0000 complete [ 2081.962217] LustreError: 95977:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2084.212295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2089.196964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2089.335340] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2089.337627] Lustre: Found index 0 for lustre-OST0000, updating log [ 2089.342265] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2089.408548] Lustre: lustre-OST0000: new disk, initializing [ 2089.413207] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2089.623497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2093.827519] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2097.298746] Lustre: Failing over lustre-OST0000 [ 2097.368764] LustreError: 97063:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2097.384335] LustreError: 97063:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2097.513396] Lustre: server umount lustre-OST0000 complete [ 2102.244737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2102.250092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2102.256767] Lustre: Skipped 3 previous similar messages [ 2106.780298] LustreError: 97264:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2106.785906] LustreError: 97264:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2106.889653] Lustre: server umount lustre-MDT0000 complete [ 2107.364271] LustreError: 94720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2107.390841] LustreError: 94720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2110.372977] LustreError: 93630:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337858 with bad export cookie 10259591072810861805 [ 2110.375641] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2110.393835] LustreError: 93630:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2120.693458] LustreError: 97466:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2120.697184] LustreError: 97466:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2120.867512] Lustre: server umount lustre-MDT0001 complete [ 2128.661943] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 19:04:36 (1763337876) [ 2144.122253] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 2153.801155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2154.199186] LustreError: 98835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.372315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2158.188559] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2168.083972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2173.081231] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2176.304975] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2184.248929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2185.172101] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2190.789465] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2193.320115] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2193.371110] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2193.376568] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2200.135501] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2200.193658] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2200.235142] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2200.315972] Lustre: lustre-OST0001: new disk, initializing [ 2200.322477] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2200.591190] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2200.593966] Lustre: Skipped 2 previous similar messages [ 2205.805307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2205.822220] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2205.946947] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2206.550236] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2214.727785] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2220.347425] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2229.494626] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.55@tcp (stopping) [ 2229.504471] Lustre: Skipped 1 previous similar message [ 2230.243039] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2230.247668] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2234.580802] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.55@tcp (stopping) [ 2234.584105] Lustre: Skipped 4 previous similar messages [ 2235.359266] LustreError: 102079:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2235.378676] LustreError: 102079:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2235.483041] Lustre: server umount lustre-MDT0001 complete [ 2236.898908] LustreError: 98835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2236.927718] LustreError: 98835:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2238.574672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2244.907964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2244.989754] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2244.993877] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2245.030672] Lustre: 98824:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2246.587963] LustreError: 102664:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2246.599570] LustreError: 102664:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2246.604709] LustreError: 98827:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2246.732632] LustreError: 102686:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.155@tcp: cfg command failed: rc = -17 [ 2246.752382] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2246.752382] [ 2249.547534] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2251.254067] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2257.828955] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 19:06:46 (1763338006) [ 2272.091462] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 2287.946799] Lustre: 104692:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2308.145444] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2311.243461] Lustre: 105826:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2394.081882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2394.090752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2394.109406] Lustre: Skipped 2 previous similar messages [ 2394.114611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2399.623465] LustreError: 106313:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2399.627000] LustreError: 106313:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2399.713080] LustreError: 98831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2399.740704] LustreError: 98831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2399.756313] Lustre: server umount lustre-MDT0000 complete [ 2402.819402] LustreError: 98815:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338151 with bad export cookie 10259591072810862736 [ 2402.826580] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2403.079813] Lustre: server umount lustre-MDT0001 complete [ 2416.493539] Lustre: server umount lustre-OST0000 complete [ 2430.479407] Lustre: server umount lustre-OST0001 complete [ 2435.264577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2442.743511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2443.628234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2450.191419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2458.188806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2459.015962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2473.607255] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 2481.695237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2481.722565] Lustre: lustre-MDT0000: reset Object Index mappings [ 2482.055150] LustreError: 109806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.159487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2482.162978] Lustre: Skipped 1 previous similar message [ 2485.482349] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2492.740158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2492.753512] Lustre: lustre-MDT0001: reset Object Index mappings [ 2496.644501] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2499.313490] Lustre: 110920:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2504.669187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2507.249951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 2509.259193] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2515.552731] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2517.576963] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2518.653050] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:227 to 0x280000400:257) [ 2518.654759] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2520.171971] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2526.394207] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2528.865923] Lustre: 112776:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2574.809181] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 19:12:02 (1763338322) [ 2600.929765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2600.936806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2600.940607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2600.940616] Lustre: Skipped 3 previous similar messages [ 2600.955344] Lustre: Skipped 5 previous similar messages [ 2606.546517] LustreError: 113127:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2606.551457] LustreError: 113127:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2606.669889] Lustre: server umount lustre-MDT0000 complete [ 2610.439489] LustreError: 109785:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338358 with bad export cookie 10259591072810909671 [ 2610.454302] LustreError: 109785:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2610.455313] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2611.168888] LustreError: 109800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.172167] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2611.205233] LustreError: 109800:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2611.225241] Lustre: Skipped 2 previous similar messages [ 2616.954261] Lustre: server umount lustre-MDT0001 complete [ 2626.883944] Lustre: server umount lustre-OST0000 complete [ 2636.636028] Lustre: server umount lustre-OST0001 complete [ 2642.668475] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 2650.878119] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 2658.394386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2663.681721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2668.778094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2674.343253] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2680.072689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2680.124818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2680.281278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2680.313691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2680.362771] Lustre: lustre-MDT0000: new disk, initializing [ 2680.433253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2680.437632] Lustre: Skipped 3 previous similar messages [ 2680.460171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2683.413408] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2689.051855] Lustre: server umount lustre-MDT0000 complete [ 2691.097397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2697.135463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2697.204564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2697.503858] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2697.516704] Lustre: 117004:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2697.530765] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2700.573079] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2704.470367] Lustre: server umount lustre-MDT0000 complete [ 2712.032233] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 19:14:20 (1763338460) [ 2713.293566] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2714.852924] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 19:14:23 (1763338463) [ 2729.245884] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 2738.490584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2743.738530] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2753.879301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2753.983672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2754.104793] Lustre: 119021:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2754.194747] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2754.197839] Lustre: Skipped 1 previous similar message [ 2754.345595] Lustre: lustre-MDT0001: new disk, initializing [ 2754.487821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2754.499865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2759.078085] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2763.477768] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2769.882210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2769.998284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2770.250845] Lustre: lustre-OST0000: new disk, initializing [ 2770.264187] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2771.416148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2771.431175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2771.585874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2776.076900] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2785.599817] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2785.663307] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2785.795620] Lustre: lustre-OST0001: new disk, initializing [ 2785.798644] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2787.483649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2787.496778] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2787.581122] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2791.627214] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 2800.514411] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2803.487763] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3144.532334] Lustre: 127822:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3144.537979] Lustre: 127822:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3144.548775] Lustre: 127822:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3144.555818] Lustre: 127822:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3144.574291] Lustre: 127822:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3144.587967] Lustre: 127822:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3373.814474] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:25:21 (1763339121) [ 3389.409275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3389.423651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3389.448963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3389.456514] Lustre: Skipped 11 previous similar messages [ 3391.981556] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3393.596994] LustreError: 128051:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3393.619910] LustreError: 128051:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 3393.809088] Lustre: server umount lustre-MDT0000 complete [ 3397.090748] LustreError: 122217:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3397.104469] LustreError: 122217:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3397.181568] LustreError: 127872:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339145 with bad export cookie 10259591072810974687 [ 3397.191265] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3397.195979] LustreError: 127872:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3402.208774] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3402.210546] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3402.217540] Lustre: Skipped 3 previous similar messages [ 3402.226954] Lustre: Skipped 5 previous similar messages [ 3403.665210] Lustre: server umount lustre-MDT0001 complete [ 3412.739364] LustreError: 128454:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3412.745207] LustreError: 128454:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 3412.848870] Lustre: server umount lustre-OST0000 complete [ 3422.907869] Lustre: server umount lustre-OST0001 complete [ 3429.824718] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 3439.172235] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 3449.282673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3455.766728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3461.483463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3467.541949] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3473.070990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3473.131528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3473.317529] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3473.348427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3473.415365] Lustre: lustre-MDT0000: new disk, initializing [ 3473.481136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3473.484027] Lustre: Skipped 5 previous similar messages [ 3473.511315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3476.566728] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 3487.000764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3487.058133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3487.119437] Lustre: 130842:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3487.177735] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3487.181167] Lustre: Skipped 1 previous similar message [ 3487.289246] Lustre: lustre-MDT0001: new disk, initializing [ 3487.409934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3487.422655] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3491.068481] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 3499.098343] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3502.979471] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3509.675878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3509.735785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3509.918603] Lustre: lustre-OST0000: new disk, initializing [ 3509.929700] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3511.134984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3511.143694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3511.231240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3515.444515] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 3522.470620] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3531.236433] Lustre: lustre-MDD0000: changelog on [ 3533.577646] Lustre: lustre-MDD0001: changelog on [ 5323.500395] Lustre: lustre-MDD0001: changelog off [ 5325.044849] Lustre: 133928:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5325.712069] Lustre: lustre-MDD0000: changelog off [ 5333.989494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5333.993138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5334.003771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5334.034243] Lustre: Skipped 1 previous similar message [ 5334.038341] Lustre: Skipped 1 previous similar message [ 5336.188549] LustreError: 140644:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5336.191320] LustreError: 140644:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5336.326527] Lustre: server umount lustre-OST0000 complete [ 5341.158265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5341.160954] LustreError: Skipped 1 previous similar message [ 5341.166134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5341.174295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5344.224332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5344.225453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5345.873735] LustreError: 140846:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5345.882030] LustreError: 140846:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5345.997459] Lustre: server umount lustre-MDT0000 complete [ 5348.776254] LustreError: 130834:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341096 with bad export cookie 10259591072811749580 [ 5348.797895] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5349.106859] Lustre: server umount lustre-MDT0001 complete [ 5357.932232] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 5360.369694] Key type lgssc unregistered [ 5360.597534] LNet: 141579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5360.601604] LNetError: 141579:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5360.615472] LNet: Removed LNI 192.168.202.155@tcp [ 5361.403415] Key type .llcrypt unregistered [ 5361.405507] Key type ._llcrypt unregistered [ 5377.748615] Key type ._llcrypt registered [ 5377.752946] Key type .llcrypt registered [ 5377.871287] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 5390.240651] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 5391.053490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5391.244748] alg: No test for adler32 (adler32-zlib) [ 5392.254924] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 5392.430426] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 5394.057527] Key type lgssc registered [ 5395.055656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5402.769075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5407.665931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5412.420382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5416.876151] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5421.753766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5421.791088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5421.799832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5422.995390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5423.040133] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5423.096336] Lustre: lustre-MDT0000: new disk, initializing [ 5423.154416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5423.175567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5425.712166] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5432.999812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5433.045202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5433.143577] Lustre: 144732:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5433.189279] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5433.193136] Lustre: Skipped 1 previous similar message [ 5433.251216] Lustre: lustre-MDT0001: new disk, initializing [ 5433.328155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5433.379879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5433.399962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5436.243067] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5442.560234] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5445.143344] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5449.964610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5450.031892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5450.212482] Lustre: lustre-OST0000: new disk, initializing [ 5450.218699] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5450.231276] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 5450.314100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5455.358648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5455.373578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5455.421744] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5455.433134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5461.818082] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5466.958583] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_module obdecho/obdecho [ 5468.413084] LustreError: 147358:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5468.417616] LustreError: 147358:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 5469.914876] LustreError: 147455:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 5479.392870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5479.396866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5479.404395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5480.934025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5480.935409] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5480.937400] Lustre: Skipped 1 previous similar message [ 5480.950947] Lustre: Skipped 1 previous similar message [ 5485.497244] LustreError: 147795:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 5485.647523] Lustre: server umount lustre-MDT0000 complete [ 5489.047246] LustreError: 144724:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341237 with bad export cookie 15852164720853295617 [ 5489.051457] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5489.052491] LustreError: 144724:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5489.165367] LustreError: 147996:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 5489.170718] LustreError: 147996:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5489.306966] Lustre: server umount lustre-MDT0001 complete [ 5502.667289] LustreError: 148197:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5502.675050] LustreError: 148197:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5502.765907] Lustre: server umount lustre-OST0000 complete [ 5518.971972] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 5527.321630] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 5534.529281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5539.772885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5544.782814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5550.044918] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5555.681566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5555.754980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5555.948719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5555.984995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5556.053991] Lustre: lustre-MDT0000: new disk, initializing [ 5556.144082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5556.158198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5559.872683] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5569.499105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5569.578453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5569.667299] Lustre: 151156:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5569.743795] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5569.762085] Lustre: Skipped 1 previous similar message [ 5569.884704] Lustre: lustre-MDT0001: new disk, initializing [ 5570.002212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5570.052935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5570.068534] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5573.482456] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5579.691472] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5582.671687] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5588.134485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5588.187914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5588.372571] Lustre: lustre-OST0000: new disk, initializing [ 5588.377576] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5588.489484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5593.433519] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5596.267668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5596.285824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5596.386193] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5599.746774] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5603.147385] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5603.371768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5606.839717] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5607.032572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5612.002557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5612.003626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5612.012028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5612.033908] Lustre: Skipped 2 previous similar messages [ 5614.647638] LustreError: 154094:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5614.650900] LustreError: 154094:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5614.756928] Lustre: server umount lustre-OST0000 complete [ 5621.219978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5621.224059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5621.239834] Lustre: Skipped 1 previous similar message [ 5621.245799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5623.752728] LustreError: 154296:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5623.757430] LustreError: 154296:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5623.846253] Lustre: server umount lustre-MDT0000 complete [ 5626.972483] LustreError: 151148:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341375 with bad export cookie 15852164720853296576 [ 5626.978970] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5626.988892] LustreError: 151148:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5627.251599] Lustre: server umount lustre-MDT0001 complete [ 5630.373075] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 20:02:58 (1763341378) [ 5631.908234] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5633.383203] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 20:03:01 (1763341381) [ 5639.411816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5639.820518] LustreError: 155170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5639.846096] LustreError: 155170:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5639.916197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5643.098588] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5645.282113] LustreError: 155170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5650.019526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5653.946270] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5658.865654] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5662.102478] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5667.880585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5672.134917] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5676.462863] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5696.480448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5696.489552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5696.501538] Lustre: Skipped 1 previous similar message [ 5696.505356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5696.509398] Lustre: Skipped 2 previous similar messages [ 5698.810196] LustreError: 157444:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5698.820828] LustreError: 157444:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5698.974480] Lustre: server umount lustre-MDT0000 complete [ 5701.532815] LustreError: 155151:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341449 with bad export cookie 15852164720853297458 [ 5701.533352] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5701.540439] LustreError: 155151:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5702.116765] LustreError: 156618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5706.143934] LustreError: 155166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5706.156264] LustreError: 155166:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5707.961331] Lustre: server umount lustre-MDT0001 complete [ 5713.354621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5724.372815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5724.577288] LustreError: 158704:0:(llog.c:1616:llog_backup()) MGC192.168.202.155@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5724.581843] Lustre: 158704:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.202.155@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5730.849526] LustreError: 158712:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5730.869189] LustreError: 158712:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5741.025414] LustreError: 158712:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5741.035713] LustreError: 158712:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 5758.665002] LustreError: 158712:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5758.672925] LustreError: 158712:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 5785.567921] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5785.586164] Lustre: 158704:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.202.155@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5792.225315] LustreError: 158711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5792.245550] LustreError: 158711:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 23 previous similar messages [ 5814.239337] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5814.242142] Lustre: 155161:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.202.155@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5826.548904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5826.555534] Lustre: Skipped 2 previous similar messages [ 5829.413868] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5831.657444] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5834.721123] LustreError: 155161:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 5836.299054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5845.985128] LustreError: 142944:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000005b5ee4 x1848997199337216/t0(0) o250->MGC192.168.202.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5849.481213] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5851.626263] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5854.422724] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5855.926385] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5859.124886] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5860.489870] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5877.218121] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5877.219146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5877.227347] Lustre: Skipped 4 previous similar messages [ 5877.227546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5877.237668] Lustre: Skipped 8 previous similar messages [ 5882.624686] LustreError: 160211:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5882.627918] LustreError: 160211:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5882.722717] Lustre: server umount lustre-MDT0000 complete [ 5886.099990] LustreError: 155149:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341634 with bad export cookie 15852164720853303968 [ 5886.101228] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5886.106809] LustreError: 155149:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5886.393700] Lustre: server umount lustre-MDT0001 complete [ 5899.974912] Lustre: server umount lustre-OST0000 complete [ 5918.456291] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 5926.966315] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 5935.367178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5942.790906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5949.321048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5955.412308] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5962.068439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5962.142647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5962.519046] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5962.559518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5962.689880] Lustre: lustre-MDT0000: new disk, initializing [ 5962.847094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5962.861756] Lustre: Skipped 1 previous similar message [ 5962.887296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5967.162541] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5976.950638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5977.006532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5977.089797] Lustre: 163562:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5977.411584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5980.701250] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 5986.999416] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5990.961963] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5997.878625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5997.973924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5998.294946] Lustre: lustre-OST0000: new disk, initializing [ 5998.304823] Lustre: Skipped 1 previous similar message [ 5998.311047] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5998.314631] Lustre: Skipped 2 previous similar messages [ 6002.929176] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6006.325235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6006.338253] Lustre: Skipped 1 previous similar message [ 6006.340922] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6006.415205] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6008.598347] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6011.861131] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6012.022107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6014.798310] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6014.938362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6016.996918] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6016.999401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6017.001615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6017.001624] Lustre: Skipped 4 previous similar messages [ 6017.009340] LustreError: Skipped 1 previous similar message [ 6017.014188] Lustre: Skipped 2 previous similar messages [ 6022.393269] LustreError: 166503:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6022.396182] LustreError: 166503:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6022.568461] Lustre: server umount lustre-OST0000 complete [ 6040.032071] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6040.288685] Lustre: server umount lustre-MDT0000 complete [ 6042.595265] LustreError: 163569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6042.608205] LustreError: 163569:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 42 previous similar messages [ 6043.979323] LustreError: 163552:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341792 with bad export cookie 15852164720853320761 [ 6043.979976] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6043.999093] LustreError: 163552:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6044.273852] Lustre: server umount lustre-MDT0001 complete [ 6052.865178] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 20:10:00 (1763341800) [ 6059.872208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6063.722644] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6070.810212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6075.160951] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6079.172489] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6082.169568] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6088.261939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6093.861884] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6098.707860] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6117.352525] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 20:11:05 (1763341865) [ 6120.419240] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6120.430908] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6120.440139] Lustre: Skipped 2 previous similar messages [ 6120.449913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6120.452584] Lustre: Skipped 10 previous similar messages [ 6126.023860] LustreError: 170040:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6126.039062] LustreError: 170040:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6126.110788] Lustre: server umount lustre-OST0000 complete [ 6135.356768] Lustre: server umount lustre-MDT0000 complete [ 6138.475090] LustreError: 167557:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341886 with bad export cookie 15852164720853321643 [ 6138.477051] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6138.490210] LustreError: 167557:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6138.763109] Lustre: server umount lustre-MDT0001 complete [ 6147.874644] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 6150.523785] Key type lgssc unregistered [ 6150.737759] LNet: 170975:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6150.751973] LNetError: 170975:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6151.783898] LNet: Removed LNI 192.168.202.155@tcp [ 6152.573390] Key type .llcrypt unregistered [ 6152.575667] Key type ._llcrypt unregistered [ 6168.485919] Key type ._llcrypt registered [ 6168.494567] Key type .llcrypt registered [ 6168.604396] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 6169.354642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6169.616504] alg: No test for adler32 (adler32-zlib) [ 6170.783717] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 6171.013493] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 6172.695185] Key type lgssc registered [ 6173.974916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6181.670179] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6181.712540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6197.407355] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6202.527302] LustreError: 172198:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.155@tcp: failed processing log, type 4: rc = -110 [ 6207.647394] LustreError: 172198:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6207.658876] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6207.671318] LustreError: MGC192.168.202.155@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6207.689487] LustreError: 172198:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6207.706717] LustreError: 172198:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -22 [ 6207.722780] LustreError: 172198:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 6207.737591] LustreError: 172198:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 6207.818076] Lustre: server umount lustre-OST0000 complete [ 6207.820213] LustreError: 172198:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6216.935139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6218.425734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6222.570758] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6227.939685] LustreError: 172701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6227.960322] LustreError: 172701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6231.534249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6232.142222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6236.296172] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6241.587727] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6245.207020] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6251.273443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6251.482801] LustreError: 174294:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6251.498684] Lustre: 174294:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.202.155@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6251.657724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6256.809985] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6260.664582] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6262.245489] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6262.249056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6262.253550] Lustre: Skipped 2 previous similar messages [ 6262.259374] Lustre: Skipped 2 previous similar messages [ 6267.366977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6267.369814] Lustre: Skipped 2 previous similar messages [ 6268.396316] LustreError: 174945:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6268.399372] LustreError: 174945:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6268.489060] Lustre: server umount lustre-MDT0000 complete [ 6272.001038] LustreError: 172682:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342020 with bad export cookie 2719093686392758923 [ 6272.007990] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6272.017900] LustreError: 172682:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6272.147912] LustreError: 175149:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6272.163426] LustreError: 175149:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6272.347915] Lustre: server umount lustre-MDT0001 complete [ 6284.946109] LustreError: 175349:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6284.949299] LustreError: 175349:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6285.092502] Lustre: server umount lustre-OST0000 complete [ 6293.272630] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 6302.230662] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 6311.450473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6316.966223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6322.626205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6328.900029] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6335.666693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6335.740979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6335.975707] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6336.005251] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6336.059356] Lustre: lustre-MDT0000: new disk, initializing [ 6336.140181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6336.174588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6339.853721] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6349.443909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6349.525177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6349.598337] Lustre: 177686:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6349.663843] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6349.668795] Lustre: Skipped 1 previous similar message [ 6349.730374] Lustre: lustre-MDT0001: new disk, initializing [ 6349.814925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6349.848707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6349.864962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6353.009228] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6359.190241] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6363.200549] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6370.265980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6370.405883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6370.782711] Lustre: lustre-OST0000: new disk, initializing [ 6370.787863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6370.865927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6376.284816] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6380.080258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6380.091017] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6380.188598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6382.619040] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6386.642429] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6386.791100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6390.989750] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6391.183694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6395.874921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6395.882360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6395.897258] Lustre: Skipped 1 previous similar message [ 6395.897659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6395.923120] Lustre: Skipped 1 previous similar message [ 6401.000109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6401.002559] Lustre: Skipped 1 previous similar message [ 6406.113342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6406.143242] Lustre: Skipped 1 previous similar message [ 6407.135150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6407.172292] LustreError: 180624:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6407.193023] LustreError: 180624:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6407.400271] Lustre: server umount lustre-OST0000 complete [ 6411.232336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6416.352133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6416.354275] Lustre: Skipped 3 previous similar messages [ 6417.203727] LustreError: 180828:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6417.209915] LustreError: 180828:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6417.434307] Lustre: server umount lustre-MDT0000 complete [ 6421.112177] LustreError: 177679:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342169 with bad export cookie 2719093686392759637 [ 6421.124485] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6421.131818] LustreError: 177679:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6421.480333] Lustre: server umount lustre-MDT0001 complete [ 6429.373247] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 20:16:17 (1763342177) [ 6444.891710] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 6447.599932] Key type lgssc unregistered [ 6447.865659] LNet: 182216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6447.884266] LNetError: 182216:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6447.902564] LNet: Removed LNI 192.168.202.155@tcp [ 6448.758291] Key type .llcrypt unregistered [ 6448.759786] Key type ._llcrypt unregistered [ 6465.244155] Key type ._llcrypt registered [ 6465.245817] Key type .llcrypt registered [ 6465.360661] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 6466.237049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6466.278560] alg: No test for adler32 (adler32-zlib) [ 6467.322906] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 6467.539362] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 6469.199243] Key type lgssc registered [ 6470.210519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6476.851659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6476.882116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6478.209424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6482.059353] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6488.548851] LustreError: 183415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6488.561311] LustreError: 183415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6490.176668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6490.620939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6493.932647] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6498.540521] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6501.664827] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6508.560716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6508.849607] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6508.851835] LustreError: 185056:0:(llog.c:1657:llog_backup()) MGC192.168.202.155@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6508.880375] Lustre: 185056:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.202.155@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6509.071458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6514.292455] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6518.943173] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6521.313885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6521.317743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6521.329431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6525.414360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6525.418205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6525.437201] Lustre: Skipped 1 previous similar message [ 6525.442443] Lustre: Skipped 1 previous similar message [ 6527.074169] LustreError: 185708:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6527.162556] Lustre: server umount lustre-MDT0000 complete [ 6530.464452] LustreError: 183396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342278 with bad export cookie 13706673332397765026 [ 6530.473191] LustreError: 183396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6530.473616] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6530.527815] LustreError: 183410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6530.530668] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6530.577505] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6530.580380] Lustre: Skipped 1 previous similar message [ 6530.618150] LustreError: 185911:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6530.630728] LustreError: 185911:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6530.761319] Lustre: server umount lustre-MDT0001 complete [ 6536.298832] LustreError: 186112:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6536.307621] LustreError: 186112:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6536.460325] Lustre: server umount lustre-OST0000 complete [ 6544.852555] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 6553.212234] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 6562.392944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6569.521635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6575.890120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6583.192144] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6590.660019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6590.803467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6591.162926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6591.194722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6591.317814] Lustre: lustre-MDT0000: new disk, initializing [ 6591.427699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6591.443500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6595.739114] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6605.212682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6605.273986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6605.341774] Lustre: 188450:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6605.378959] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6605.381022] Lustre: Skipped 1 previous similar message [ 6605.387162] LustreError: 189217:0:(llog.c:1657:llog_backup()) MGC192.168.202.155@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6605.395996] LustreError: 189217:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 6605.403752] Lustre: 189217:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.202.155@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6605.419817] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6605.423852] Lustre: Skipped 1 previous similar message [ 6605.515720] Lustre: lustre-MDT0001: new disk, initializing [ 6605.641692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6605.680741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6605.702329] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6608.953386] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6614.763138] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6617.661761] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6623.333373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6623.376822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6623.639287] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6623.641420] Lustre: Skipped 1 previous similar message [ 6623.650729] LustreError: 190173:0:(llog.c:1657:llog_backup()) MGC192.168.202.155@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6623.661132] LustreError: 190173:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 6623.666518] Lustre: 190173:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.202.155@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 6623.672666] Lustre: 190173:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6623.696877] Lustre: lustre-OST0000: new disk, initializing [ 6623.699714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6623.792720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6628.752044] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6633.004402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6633.011582] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6633.124319] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6634.595939] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6638.021528] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6638.171745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6641.917849] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6642.099929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6648.803172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6648.808085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6648.829938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6648.842164] Lustre: Skipped 1 previous similar message [ 6649.927133] LustreError: 191389:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6649.929339] LustreError: 191389:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6650.134617] Lustre: server umount lustre-OST0000 complete [ 6653.924948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6653.946270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6653.949139] Lustre: Skipped 1 previous similar message [ 6664.160704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6664.166638] Lustre: Skipped 4 previous similar messages [ 6668.266813] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6668.433267] LustreError: 191592:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6668.446918] LustreError: 191592:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6668.579444] Lustre: server umount lustre-MDT0000 complete [ 6669.284100] LustreError: 190537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6669.301962] LustreError: 190537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6672.046286] LustreError: 188443:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342420 with bad export cookie 13706673332397765740 [ 6672.052082] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6672.067114] LustreError: 188443:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6672.670994] Lustre: server umount lustre-MDT0001 complete [ 6683.612955] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 20:20:31 (1763342431) [ 6700.193310] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 6710.036874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6710.590805] LustreError: 193162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6710.659816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6714.602561] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6715.876743] LustreError: 193162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6723.727944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6728.809917] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6731.794823] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6739.783947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6747.533298] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6757.289380] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6757.370899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6757.574893] Lustre: lustre-OST0001: new disk, initializing [ 6757.583430] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6757.718943] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6757.723446] Lustre: Skipped 2 previous similar messages [ 6763.441758] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6763.619872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6763.637946] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6763.747560] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6773.910830] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6780.709172] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6787.826641] Lustre: DEBUG MARKER: ADD OST3 [ 6792.668229] loop: module loaded [ 6792.780668] loop0: detected capacity change from 0 to 204800000 [ 6792.991155] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6793.013082] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6793.038781] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6793.167457] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6799.410607] loop0: detected capacity change from 0 to 204800000 [ 6800.244278] Lustre: DEBUG MARKER: STOP OST3 [ 6805.816584] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6805.867203] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6806.047958] Lustre: lustre-OST0003: new disk, initializing [ 6806.051675] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 6807.852481] Lustre: *** cfs_fail_loc=2109, val=2*** [ 6808.029738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 6808.042809] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 6808.081324] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 6809.749403] Lustre: Failing over lustre-OST0003 [ 6809.854130] LustreError: 197781:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 6809.856976] LustreError: 197781:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6809.888146] LustreError: 197593:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 6809.953879] Lustre: server umount lustre-OST0003 complete [ 6812.229797] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6812.447318] LustreError: 194617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6812.483253] LustreError: 194617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6813.162714] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6813.166452] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6813.200502] Lustre: Skipped 2 previous similar messages [ 6821.178920] Lustre: DEBUG MARKER: START OST3 again [ 6827.273330] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6827.566348] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 6827.571648] Lustre: Skipped 1 previous similar message [ 6827.601698] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 6827.739390] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6828.964800] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 6828.966731] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6832.645134] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6852.431360] Lustre: Failing over lustre-OST0003 [ 6852.493232] LustreError: 199763:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 6852.567881] LustreError: 199763:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6852.654239] Lustre: server umount lustre-OST0003 complete [ 6853.105758] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6853.114971] LustreError: 194618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6853.118769] Lustre: Skipped 1 previous similar message [ 6853.129200] LustreError: 194618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 6872.544445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6872.562987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6875.826663] Lustre: server umount lustre-MDT0000 complete [ 6879.846573] LustreError: 193144:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342628 with bad export cookie 13706673332397766622 [ 6879.854352] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6879.857700] LustreError: 193144:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6886.088910] LustreError: 200239:0:(obd_class.h:479:obd_check_dev()) Device 33 not setup [ 6886.091976] LustreError: 200239:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6886.258871] Lustre: server umount lustre-MDT0001 complete [ 6889.744809] Lustre: server umount lustre-OST0000 complete [ 6893.461319] Lustre: server umount lustre-OST0001 complete [ 6899.985557] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 6907.496079] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 6916.007538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6922.285846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6927.449286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6933.501897] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6940.451662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6940.559733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6940.809848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6940.832343] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6940.891242] Lustre: lustre-MDT0000: new disk, initializing [ 6940.990132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6945.031607] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6955.498655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6955.630939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6955.792532] Lustre: 202828:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6956.225317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6956.235422] Lustre: Skipped 1 previous similar message [ 6956.294723] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6961.520118] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6969.722354] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6973.837331] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6981.098299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6981.204772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6981.476411] Lustre: lustre-OST0000: new disk, initializing [ 6981.484428] Lustre: Skipped 1 previous similar message [ 6981.493272] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6981.507304] Lustre: Skipped 2 previous similar messages [ 6983.085648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6983.089324] Lustre: Skipped 1 previous similar message [ 6983.117506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6983.219824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6987.745344] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 6994.814830] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6998.830420] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6999.039660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7002.929707] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7003.148297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7008.741337] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7008.752051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7008.762416] Lustre: Skipped 6 previous similar messages [ 7008.775752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7008.783019] Lustre: Skipped 9 previous similar messages [ 7011.003497] LustreError: 205764:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7011.017801] LustreError: 205764:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 7011.162574] Lustre: server umount lustre-OST0000 complete [ 7017.952428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7017.963852] LustreError: Skipped 1 previous similar message [ 7021.449145] Lustre: server umount lustre-MDT0000 complete [ 7024.100667] LustreError: 202835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7024.139697] LustreError: 202835:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 7024.941308] LustreError: 202819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342773 with bad export cookie 13706673332397768918 [ 7024.951961] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7025.283713] Lustre: server umount lustre-MDT0001 complete [ 7028.989578] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 20:26:17 (1763342777) [ 7044.287563] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 7053.386614] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 7062.321412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7069.611828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7076.337638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7083.692506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7090.366326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7090.440975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7090.673865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7090.729397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7090.818367] Lustre: lustre-MDT0000: new disk, initializing [ 7090.920076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7094.774194] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7105.202968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7105.261633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7105.348566] Lustre: 209012:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7105.644794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7109.691885] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7117.399150] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7120.680276] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7126.552486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7126.607370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7128.799885] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7132.577381] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7141.460650] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7146.042924] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7146.323803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7150.473902] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7150.674777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7154.662441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7154.664414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7154.671466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7154.671475] Lustre: Skipped 3 previous similar messages [ 7154.681510] Lustre: Skipped 4 previous similar messages [ 7158.653457] LustreError: 211953:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7158.657924] LustreError: 211953:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7158.862383] Lustre: server umount lustre-OST0000 complete [ 7169.003331] Lustre: server umount lustre-MDT0000 complete [ 7170.017872] LustreError: 210883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7170.046198] LustreError: 210883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7173.391988] LustreError: 210759:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342921 with bad export cookie 13706673332397769800 [ 7173.394958] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7173.411471] LustreError: 210759:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7173.886922] Lustre: server umount lustre-MDT0001 complete [ 7183.243085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7190.150417] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7199.507810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7204.087324] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7209.734856] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7213.410538] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7220.003903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7220.399570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7220.405039] Lustre: Skipped 6 previous similar messages [ 7225.477471] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7230.217865] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7241.187511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7241.206886] LustreError: Skipped 1 previous similar message [ 7246.969870] Lustre: server umount lustre-OST0000 complete [ 7265.247624] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7265.415733] Lustre: server umount lustre-MDT0000 complete [ 7269.187975] LustreError: 212817:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343017 with bad export cookie 13706673332397770682 [ 7269.198576] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7269.215810] LustreError: 212817:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7269.719928] Lustre: server umount lustre-MDT0001 complete [ 7368.297876] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 7370.981300] Key type lgssc unregistered [ 7371.252369] LNet: 216147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7371.264839] LNetError: 216147:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7371.291794] LNet: Removed LNI 192.168.202.155@tcp [ 7372.133185] Key type .llcrypt unregistered [ 7372.134808] Key type ._llcrypt unregistered [ 7394.303523] Key type ._llcrypt registered [ 7394.305904] Key type .llcrypt registered [ 7394.420612] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 7408.926473] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 7410.033956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7410.178902] alg: No test for adler32 (adler32-zlib) [ 7411.230489] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 7411.420364] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 7413.119162] Key type lgssc registered [ 7414.112363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7422.878352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7429.290690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7435.980874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7442.049752] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7448.259452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7448.308873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7448.330657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7449.550398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7449.594435] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7449.717785] Lustre: lustre-MDT0000: new disk, initializing [ 7449.807049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7449.831888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7453.553483] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7463.160479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7463.235853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7463.302669] Lustre: 219476:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7463.362638] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7463.370415] Lustre: Skipped 1 previous similar message [ 7463.458287] Lustre: lustre-MDT0001: new disk, initializing [ 7463.577742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7463.620727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7463.628495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7467.373755] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7474.728621] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7478.013348] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7484.681569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7484.774177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7485.154576] Lustre: lustre-OST0000: new disk, initializing [ 7485.168537] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7485.324042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7491.452222] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7492.187307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7492.199339] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7492.306866] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7499.530984] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7504.006761] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7504.303592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7508.298706] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7508.502504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7513.060652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7513.068763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7513.090835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7513.096068] Lustre: Skipped 1 previous similar message [ 7516.600611] LustreError: 222418:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7516.755626] Lustre: server umount lustre-OST0000 complete [ 7523.299543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7523.317190] Lustre: Skipped 1 previous similar message [ 7523.321203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7523.333185] Lustre: Skipped 1 previous similar message [ 7526.777934] LustreError: 222620:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7526.792031] LustreError: 222620:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7527.007379] Lustre: server umount lustre-MDT0000 complete [ 7530.739034] LustreError: 219469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343278 with bad export cookie 12124848897625132663 [ 7530.755917] LustreError: 219469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7530.758135] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7530.969170] LustreError: 222821:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7530.976954] LustreError: 222821:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7531.244595] Lustre: server umount lustre-MDT0001 complete [ 7548.488918] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 7557.413248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7557.821915] LustreError: 223997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7557.838158] LustreError: 223997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7557.863639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7561.133521] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7563.237097] LustreError: 223997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7568.354737] LustreError: 223997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7568.756801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7569.111062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7572.765198] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7575.548202] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7581.502374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7581.768665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7586.892656] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7594.051492] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7594.142346] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7594.293169] Lustre: lustre-OST0001: new disk, initializing [ 7594.301529] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7596.465085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7596.475929] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7596.577027] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7599.739931] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7609.393401] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7615.922555] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7635.484984] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 20:36:23 (1763343383) [ 7641.059932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7641.073394] LustreError: Skipped 1 previous similar message [ 7641.080158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7641.088788] Lustre: Skipped 1 previous similar message [ 7641.094587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7643.714465] LustreError: 227392:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7643.720662] LustreError: 227392:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7643.820455] Lustre: server umount lustre-MDT0000 complete [ 7647.001470] LustreError: 223978:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343395 with bad export cookie 12124848897625133545 [ 7647.004725] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7647.022629] LustreError: 223978:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7647.348374] Lustre: server umount lustre-MDT0001 complete [ 7660.623192] LustreError: 227794:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7660.638051] LustreError: 227794:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 7660.688235] Lustre: server umount lustre-OST0000 complete [ 7663.583189] Lustre: 217687:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763343395/real 1763343395] req@ffff8c71cbf39500 x1848999316401792/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763343411 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7663.602137] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7663.612405] Lustre: Skipped 3 previous similar messages [ 7664.822857] Lustre: server umount lustre-OST0001 complete [ 7669.769207] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 7677.669250] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 7686.087659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7691.465625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7696.633347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7702.024288] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7707.961702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7708.062204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7708.258309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7708.297254] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7708.378100] Lustre: lustre-MDT0000: new disk, initializing [ 7708.476868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7708.479238] Lustre: Skipped 1 previous similar message [ 7708.499652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7712.110424] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7723.000131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7723.071273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7723.147427] Lustre: 230183:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7723.228956] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7723.232135] Lustre: Skipped 1 previous similar message [ 7723.281572] Lustre: lustre-MDT0001: new disk, initializing [ 7723.429303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7723.438755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7726.887980] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7734.104147] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7737.448616] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7743.797389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7743.877448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7744.109239] Lustre: lustre-OST0000: new disk, initializing [ 7744.122189] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7744.207803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7744.212148] Lustre: Skipped 1 previous similar message [ 7745.214856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7745.220686] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7745.355703] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7748.629767] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7754.858430] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7758.438944] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7758.662707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7761.914237] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7762.114577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7765.989419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7765.991487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7765.993239] LustreError: Skipped 1 previous similar message [ 7765.994592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7765.994597] Lustre: Skipped 4 previous similar messages [ 7765.998839] Lustre: Skipped 1 previous similar message [ 7769.666554] LustreError: 233126:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7769.681745] LustreError: 233126:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 7769.776583] Lustre: server umount lustre-OST0000 complete [ 7774.689053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7774.691915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7774.707513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7774.711208] Lustre: Skipped 1 previous similar message [ 7778.762672] Lustre: server umount lustre-MDT0000 complete [ 7781.344106] LustreError: 230195:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7781.356278] LustreError: 230195:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7781.452611] LustreError: 231941:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343529 with bad export cookie 12124848897625135687 [ 7781.453851] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7781.467139] LustreError: 231941:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7781.703550] Lustre: server umount lustre-MDT0001 complete [ 7789.093532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7789.650720] LustreError: 234006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7793.119929] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7799.280500] LustreError: 234005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7799.297244] LustreError: 234005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7801.045539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7805.723116] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7811.200458] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7814.198865] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7819.551859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7819.913810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7819.921500] Lustre: Skipped 2 previous similar messages [ 7825.305314] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 7830.897386] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7974.886427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7974.890868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7974.900737] Lustre: Skipped 1 previous similar message [ 7974.912117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7974.917711] Lustre: Skipped 2 previous similar messages [ 7980.216173] LustreError: 236290:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7980.223978] LustreError: 236290:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7980.336559] Lustre: server umount lustre-OST0000 complete [ 7986.144745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7986.158795] LustreError: Skipped 1 previous similar message [ 7990.194717] Lustre: server umount lustre-MDT0000 complete [ 7993.314292] LustreError: 234006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7993.325353] LustreError: 234006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7993.981585] LustreError: 233985:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343742 with bad export cookie 12124848897625136569 [ 7993.994315] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7994.002395] LustreError: 233985:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7994.371405] Lustre: server umount lustre-MDT0001 complete [ 8098.552904] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 8102.232499] Key type lgssc unregistered [ 8102.615800] LNet: 237261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8102.630992] LNetError: 237261:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8102.654870] LNet: Removed LNI 192.168.202.155@tcp [ 8103.733282] Key type .llcrypt unregistered [ 8103.735774] Key type ._llcrypt unregistered [ 8133.741923] Key type ._llcrypt registered [ 8133.743432] Key type .llcrypt registered [ 8133.849438] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 8149.721410] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 8150.945277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8151.043327] alg: No test for adler32 (adler32-zlib) [ 8152.099824] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 8152.397395] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 8154.127257] Key type lgssc registered [ 8155.185706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8163.326198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8169.806173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8175.638737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8181.860221] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8199.505253] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 8211.460980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8211.540304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8211.565634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8212.782565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8212.826268] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8212.931884] Lustre: lustre-MDT0000: new disk, initializing [ 8213.063547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8213.088532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8217.178387] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8228.477766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8228.550307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8228.637995] Lustre: 241293:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8228.724978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8228.728117] Lustre: Skipped 1 previous similar message [ 8228.780471] Lustre: lustre-MDT0001: new disk, initializing [ 8228.859976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8228.906520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8228.927186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8232.526560] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8237.331901] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8243.852563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8243.912516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8244.184095] Lustre: lustre-OST0000: new disk, initializing [ 8244.187596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8244.329550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8248.994662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8249.006978] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8249.215157] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8250.657427] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8261.624285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8261.693844] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8261.834853] Lustre: lustre-OST0001: new disk, initializing [ 8261.837866] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8261.949944] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8268.346852] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8268.380834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8268.384582] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8268.532157] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8279.286747] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8290.340329] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8305.637234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8305.641690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8305.654239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8309.733688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8309.736941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8309.746026] Lustre: Skipped 2 previous similar messages [ 8309.760880] Lustre: Skipped 2 previous similar messages [ 8310.865834] LustreError: 244816:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8311.012369] Lustre: server umount lustre-MDT0000 complete [ 8314.736545] LustreError: 242894:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344062 with bad export cookie 8097344276630184000 [ 8314.743785] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8314.753365] LustreError: 242894:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8314.848298] LustreError: 241297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8314.849114] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8314.877080] LustreError: 241297:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8314.920745] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8314.927829] Lustre: Skipped 2 previous similar messages [ 8314.945040] LustreError: 245018:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 8314.954476] LustreError: 245018:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8315.164240] Lustre: server umount lustre-MDT0001 complete [ 8321.237142] LustreError: 245218:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8321.261087] LustreError: 245218:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8321.364421] Lustre: server umount lustre-OST0000 complete [ 8325.079501] LustreError: 245419:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8325.082839] LustreError: 245419:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8325.235635] Lustre: server umount lustre-OST0001 complete [ 8339.009885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8347.748705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8348.807733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8357.787504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8365.921752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8366.978647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8385.350778] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 8395.652131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8395.678443] Lustre: lustre-MDT0000: reset Object Index mappings [ 8396.224397] LustreError: 248733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8396.370241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8400.433467] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8401.386088] LustreError: 248733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8406.500387] LustreError: 248733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8408.855218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8408.888321] Lustre: lustre-MDT0001: reset Object Index mappings [ 8409.422651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8413.072906] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8416.387344] Lustre: 249843:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8423.624322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8429.351971] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8435.301653] LustreError: 250199:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8437.930529] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8438.203396] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8438.210956] Lustre: Skipped 1 previous similar message [ 8440.965330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8440.976763] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8443.927115] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8451.808324] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8455.646822] Lustre: 251693:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8465.891907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8465.894721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8465.918210] Lustre: Skipped 1 previous similar message [ 8465.923532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8467.858772] LustreError: 251798:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8467.863916] LustreError: 251798:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8468.096355] Lustre: server umount lustre-MDT0000 complete [ 8472.035240] LustreError: 249450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8472.061771] LustreError: 249450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8472.784279] LustreError: 248714:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344220 with bad export cookie 8097344276630190783 [ 8472.822812] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8472.838464] LustreError: 248714:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8473.168608] Lustre: server umount lustre-MDT0001 complete [ 8488.224874] LustreError: 252200:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 8488.227810] LustreError: 252200:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 8488.290251] Lustre: server umount lustre-OST0000 complete [ 8501.196860] Lustre: server umount lustre-OST0001 complete [ 8519.464825] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 20:51:07 (1763344267) [ 8534.753585] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 8545.141588] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 8555.931747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8563.789153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8573.281758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8580.583041] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8586.829596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8586.890095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8587.120421] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8587.163815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8587.235968] Lustre: lustre-MDT0000: new disk, initializing [ 8587.329275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8587.349835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8591.374581] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8602.441969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8602.500433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8602.563995] Lustre: 255928:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8602.623620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8602.628227] Lustre: Skipped 1 previous similar message [ 8602.680612] Lustre: lustre-MDT0001: new disk, initializing [ 8602.800561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8602.810714] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8606.478527] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8614.426098] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8618.679917] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8625.374221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8625.441373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8625.718505] Lustre: lustre-OST0000: new disk, initializing [ 8625.729749] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8626.840138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8626.856272] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8626.998204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8630.947508] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8636.740050] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8640.303251] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8640.486420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8643.650826] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8643.886154] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8647.653878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8647.654273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8647.656454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8647.656465] Lustre: Skipped 4 previous similar messages [ 8647.656630] LustreError: Skipped 1 previous similar message [ 8647.672520] Lustre: Skipped 4 previous similar messages [ 8651.799449] LustreError: 258870:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8651.801601] LustreError: 258870:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 8651.968316] Lustre: server umount lustre-OST0000 complete [ 8657.894673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8657.915884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8657.923704] Lustre: Skipped 2 previous similar messages [ 8661.930935] Lustre: server umount lustre-MDT0000 complete [ 8663.008614] LustreError: 255939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8663.030141] LustreError: 255939:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8664.978790] LustreError: 255920:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344413 with bad export cookie 8097344276630192883 [ 8664.986551] LustreError: 255920:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8664.987141] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8665.252862] Lustre: server umount lustre-MDT0001 complete [ 8681.212623] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 8689.468765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8689.912380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8689.915633] Lustre: Skipped 2 previous similar messages [ 8693.100361] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8695.275225] LustreError: 260450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8695.311502] LustreError: 260450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8700.667148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8704.601863] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8707.087264] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8712.925775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8718.337664] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8725.017766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8725.069272] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8725.173729] Lustre: lustre-OST0001: new disk, initializing [ 8725.178198] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8726.633480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8726.643512] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8726.816584] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8729.530537] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8737.092982] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8740.607921] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8751.465635] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8751.773488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8757.222594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8757.233413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8757.253615] Lustre: Skipped 1 previous similar message [ 8757.263039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8763.251718] LustreError: 263953:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8763.257732] LustreError: 263953:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 8763.342952] Lustre: server umount lustre-MDT0000 complete [ 8766.496838] LustreError: 263459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344514 with bad export cookie 8097344276630193765 [ 8766.498553] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8766.514294] LustreError: 263459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8766.805344] Lustre: server umount lustre-MDT0001 complete [ 8780.508344] Lustre: server umount lustre-OST0000 complete [ 8793.554568] Lustre: server umount lustre-OST0001 complete [ 8805.858734] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 8814.355753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8814.672951] LustreError: 265735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8814.686384] LustreError: 265735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8817.967860] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8825.433703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8825.916802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8825.923649] Lustre: Skipped 4 previous similar messages [ 8829.332766] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8831.809391] Lustre: 266843:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8837.810258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8839.097385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 8843.284083] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8851.312301] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8852.719729] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 8856.042449] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 8861.865307] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8864.966758] Lustre: 268698:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8874.445345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 8874.566215] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 8881.690742] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 20:57:10 (1763344630) [ 8897.505446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8897.509898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8897.514306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8897.515714] Lustre: Skipped 4 previous similar messages [ 8897.519241] Lustre: Skipped 7 previous similar messages [ 8900.519017] LustreError: 269403:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8900.527266] LustreError: 269403:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 8902.594195] Lustre: server umount lustre-MDT0000 complete [ 8905.065846] LustreError: 266845:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344653 with bad export cookie 8097344276630200086 [ 8905.072140] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8905.074239] LustreError: 266845:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8911.459434] Lustre: server umount lustre-MDT0001 complete [ 8920.855459] Lustre: server umount lustre-OST0000 complete [ 8930.066734] Lustre: server umount lustre-OST0001 complete [ 8935.521120] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 8943.252954] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 8950.811418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8956.430182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8962.261953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8968.873425] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8984.377198] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 8991.849671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8991.913567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8992.090990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8992.112908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8992.159449] Lustre: lustre-MDT0000: new disk, initializing [ 8992.239189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8996.509467] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9006.121727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9006.184260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9006.245457] Lustre: 272895:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9006.442954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9009.881910] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9019.966664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9020.013285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9022.200751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9024.903953] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9033.433546] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9033.502991] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9035.285045] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9037.982322] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9045.996436] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9049.637939] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9049.641402] Lustre: Skipped 1 previous similar message [ 9066.540596] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 21:00:14 (1763344814) [ 9071.084236] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9071.091323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9071.098114] Lustre: Skipped 6 previous similar messages [ 9071.100404] Lustre: Skipped 8 previous similar messages [ 9074.602388] LustreError: 276506:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 9074.606074] LustreError: 276506:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 9074.681769] Lustre: server umount lustre-MDT0000 complete [ 9076.198398] LustreError: 275281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9076.212920] LustreError: 275281:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 9078.270100] LustreError: 272887:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344826 with bad export cookie 8097344276630207975 [ 9078.273364] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9078.283202] LustreError: 272887:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9078.578872] Lustre: server umount lustre-MDT0001 complete [ 9092.490309] Lustre: server umount lustre-OST0000 complete [ 9105.227170] Lustre: server umount lustre-OST0001 complete [ 9122.616640] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 9131.800238] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 9140.882085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9146.540286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9152.770755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9158.678840] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9166.579692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9166.647817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9166.815790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9166.847911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9166.851401] Lustre: Skipped 4 previous similar messages [ 9166.909651] Lustre: lustre-MDT0000: new disk, initializing [ 9166.912330] Lustre: Skipped 3 previous similar messages [ 9167.006256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9167.011499] Lustre: Skipped 6 previous similar messages [ 9167.038425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9167.045982] Lustre: Skipped 3 previous similar messages [ 9170.823357] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9180.328737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9180.386874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9180.441604] Lustre: 279969:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9180.713435] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9180.719306] Lustre: Skipped 2 previous similar messages [ 9184.401936] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9191.663930] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9195.044614] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9200.927547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9200.976422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9202.720356] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9206.863504] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9214.063659] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9216.153367] Lustre: Mounted lustre-client [ 9216.919356] LustreError: 282475:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8c72f9053800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 9217.021192] Lustre: Unmounted lustre-client [ 9226.726329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9226.732949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9226.733858] Lustre: Skipped 2 previous similar messages [ 9226.744683] Lustre: Skipped 3 previous similar messages [ 9232.066895] Lustre: server umount lustre-OST0000 complete [ 9236.960220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9242.039553] Lustre: server umount lustre-MDT0000 complete [ 9246.033900] LustreError: 281724:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344994 with bad export cookie 8097344276630209501 [ 9246.043099] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9246.048097] LustreError: 281724:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9246.494995] Lustre: server umount lustre-MDT0001 complete [ 9259.293155] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 9262.574923] Key type lgssc unregistered [ 9269.087320] LNet: 1 local NIs in recovery (showing 1): 192.168.202.155@tcp [ 9269.154096] LNet: 283753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9269.164243] LNetError: 283753:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9349.102567] LNet: Removed LNI 192.168.202.155@tcp [ 9349.837168] Key type .llcrypt unregistered [ 9349.843465] Key type ._llcrypt unregistered [ 9365.264106] Key type ._llcrypt registered [ 9365.265761] Key type .llcrypt registered [ 9365.354861] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [ 9379.234203] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [ 9380.091295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9380.210401] alg: No test for adler32 (adler32-zlib) [ 9381.292652] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 9381.555348] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [ 9383.207193] Key type lgssc registered [ 9384.463099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9392.626324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9398.306704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9404.008199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9409.913310] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9412.053728] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 21:05:59 (1763345159) [ 9417.603072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9417.640597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9417.654854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9418.861865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9418.895422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9418.942941] Lustre: lustre-MDT0000: new disk, initializing [ 9419.031435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9419.047724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9422.619912] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9432.262852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9432.317995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9432.384141] Lustre: 286948:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9432.446399] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9432.449257] Lustre: Skipped 1 previous similar message [ 9432.517721] Lustre: lustre-MDT0001: new disk, initializing [ 9432.629618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9432.696793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9432.720710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9436.103196] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9442.123916] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9444.988242] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9450.951798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9451.002495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9451.328892] Lustre: lustre-OST0000: new disk, initializing [ 9451.341916] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9451.438677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9456.263806] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_default_debug -1 all [ 9457.238891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9457.248152] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9457.341785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9463.093930] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9472.994391] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9473.002374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9473.033800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9476.225120] LustreError: 289390:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9476.435145] Lustre: server umount lustre-OST0000 complete [ 9483.232991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9483.252113] Lustre: Skipped 1 previous similar message [ 9483.265532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9483.268440] Lustre: Skipped 1 previous similar message [ 9486.310041] LustreError: 289591:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9486.317521] LustreError: 289591:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9486.455915] Lustre: server umount lustre-MDT0000 complete [ 9490.144764] LustreError: 286939:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345238 with bad export cookie 2349281513572535417 [ 9490.153045] LustreError: MGC192.168.202.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9490.164752] LustreError: 286939:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9490.308570] LustreError: 289793:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9490.315629] LustreError: 289793:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9490.531760] Lustre: server umount lustre-MDT0001 complete [ 9499.753405] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 9502.339201] Key type lgssc unregistered [ 9502.663295] LNet: 290325:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9502.673329] LNetError: 290325:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9503.720369] LNet: Removed LNI 192.168.202.155@tcp [ 9504.420768] Key type .llcrypt unregistered [ 9504.422655] Key type ._llcrypt unregistered [ 9590.609144] Key type ._llcrypt registered [ 9590.610810] Key type .llcrypt registered [ 9590.712837] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 9591.002556] Key type .llcrypt unregistered [ 9591.010192] Key type ._llcrypt unregistered [ 9661.967828] Key type ._llcrypt registered [ 9661.969710] Key type .llcrypt registered [ 9662.076090] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 9662.316283] Key type .llcrypt unregistered [ 9662.318045] Key type ._llcrypt unregistered [ 9690.736039] Key type ._llcrypt registered [ 9690.737292] Key type .llcrypt registered [ 9690.921050] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 9691.187298] Key type .llcrypt unregistered [ 9691.189874] Key type ._llcrypt unregistered [ 9722.834335] Key type ._llcrypt registered [ 9722.836571] Key type .llcrypt registered [ 9722.915232] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 9723.267364] Key type .llcrypt unregistered [ 9723.273595] Key type ._llcrypt unregistered [ 9805.049361] Key type ._llcrypt registered [ 9805.051383] Key type .llcrypt registered [ 9805.169106] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 9805.446865] Key type .llcrypt unregistered [ 9805.450928] Key type ._llcrypt unregistered [ 9834.823110] Key type ._llcrypt registered [ 9834.832614] Key type .llcrypt registered [ 9834.951320] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 9835.193617] Key type .llcrypt unregistered [ 9835.199551] Key type ._llcrypt unregistered [ 9863.467135] Key type ._llcrypt registered [ 9863.473189] Key type .llcrypt registered [ 9863.561979] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 9863.829379] Key type .llcrypt unregistered [ 9863.831073] Key type ._llcrypt unregistered [ 9888.445973] Key type ._llcrypt registered [ 9888.447579] Key type .llcrypt registered [ 9888.530966] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 9888.770317] Key type .llcrypt unregistered [ 9888.771904] Key type ._llcrypt unregistered [ 9913.781958] Key type ._llcrypt registered [ 9913.783712] Key type .llcrypt registered [ 9913.880841] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 9914.079276] Key type .llcrypt unregistered [ 9914.081040] Key type ._llcrypt unregistered [ 9940.446592] Key type ._llcrypt registered [ 9940.448434] Key type .llcrypt registered [ 9940.544401] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 9940.756444] Key type .llcrypt unregistered [ 9940.761494] Key type ._llcrypt unregistered [ 9991.969129] Key type ._llcrypt registered [ 9991.973855] Key type .llcrypt registered [ 9992.074459] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [ 9992.356537] Key type .llcrypt unregistered [ 9992.361031] Key type ._llcrypt unregistered [10019.839467] Key type ._llcrypt registered [10019.841239] Key type .llcrypt registered [10019.962542] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [10020.229700] Key type .llcrypt unregistered [10020.234152] Key type ._llcrypt unregistered [10046.077374] Key type ._llcrypt registered [10046.078931] Key type .llcrypt registered [10046.205067] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [10046.510337] Key type .llcrypt unregistered [10046.518172] Key type ._llcrypt unregistered [10105.566836] Key type ._llcrypt registered [10105.568598] Key type .llcrypt registered [10105.678761] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing set_hostid [10119.668121] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing load_modules_local [10120.645095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10120.738204] alg: No test for adler32 (adler32-zlib) [10121.848409] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [10122.128700] LNet: Added LNI 192.168.202.155@tcp [8/256/0/180] [10123.815219] Key type lgssc registered [10125.148593] Lustre: Echo OBD driver; http://www.lustre.org/ [10134.138067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10141.043727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10147.115362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10153.477789] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10155.337969] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9924 sec ========== 21:18:23 (1763345903) [10157.052793] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:18:25 (1763345905) === [10160.113334] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:18:28 (1763345908) === [10198.755822] Lustre: DEBUG MARKER: oleg255-server.virtnet: executing unload_modules_local [10201.664684] Key type lgssc unregistered [10201.951777] LNet: 312325:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10201.961548] LNetError: 312325:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10201.983219] LNet: Removed LNI 192.168.202.155@tcp [10202.880463] Key type .llcrypt unregistered [10202.882879] Key type ._llcrypt unregistered