[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 488858661 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003279] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.006010] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011152] LSM: Security Framework initializing [ 0.013035] Yama: becoming mindful. [ 0.014044] SELinux: Initializing. [ 0.016043] *** VALIDATE selinux *** [ 0.025028] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029643] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033004] *** VALIDATE tmpfs *** [ 0.034493] *** VALIDATE proc *** [ 0.035277] *** VALIDATE cgroup *** [ 0.036011] *** VALIDATE cgroup2 *** [ 0.038142] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040101] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042030] Spectre V2 : User space: Vulnerable [ 0.043010] Speculative Store Bypass: Vulnerable [ 0.045607] debug: unmapping init [mem 0xffffffff8de59000-0xffffffff8de60fff] [ 0.047905] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048704] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049028] ... version: 2 [ 0.050015] ... bit width: 48 [ 0.051013] ... generic registers: 4 [ 0.052016] ... value mask: 0000ffffffffffff [ 0.053017] ... max period: 00007fffffffffff [ 0.054016] ... fixed-purpose events: 3 [ 0.055014] ... event mask: 000000070000000f [ 0.056318] rcu: Hierarchical SRCU implementation. [ 0.058459] smp: Bringing up secondary CPUs ... [ 0.059626] x86: Booting SMP configuration: [ 0.060028] .... node #0, CPUs: #1 #2 #3 [ 0.063671] smp: Brought up 1 node, 4 CPUs [ 0.065016] smpboot: Max logical packages: 1 [ 0.066024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.241029] node 0 deferred pages initialised in 172ms [ 0.245060] devtmpfs: initialized [ 0.246279] x86/mm: Memory block size: 128MB [ 0.250000] gcov: version magic: 0x41383552 [ 0.252446] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.257151] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.260426] pinctrl core: initialized pinctrl subsystem [ 0.264308] [ 0.265011] ************************************************************* [ 0.267015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.271020] ** ** [ 0.273014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.275016] ** ** [ 0.278021] ** This means that this kernel is built to expose internal ** [ 0.280016] ** IOMMU data structures, which may compromise security on ** [ 0.282014] ** your system. ** [ 0.284020] ** ** [ 0.286026] ** If you see this message and you are not debugging the ** [ 0.290023] ** kernel, report this immediately to your vendor! ** [ 0.293021] ** ** [ 0.295019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.298025] ************************************************************* [ 0.300869] NET: Registered protocol family 16 [ 0.303525] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.306079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.308088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.313136] cpuidle: using governor menu [ 0.314797] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.316550] PCI: Using configuration type 1 for base access [ 0.318162] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.326128] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.328029] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.331318] cryptd: max_cpu_qlen set to 1000 [ 0.335310] ACPI: Added _OSI(Module Device) [ 0.337015] ACPI: Added _OSI(Processor Device) [ 0.339013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.341014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.345920] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.352336] ACPI: Interpreter enabled [ 0.355075] ACPI: PM: (supports S0 S3 S4 S5) [ 0.357014] ACPI: Using IOAPIC for interrupt routing [ 0.359113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.363401] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.373245] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.377042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.380018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.383072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.388512] acpiphp: Slot [2] registered [ 0.390152] acpiphp: Slot [5] registered [ 0.391340] acpiphp: Slot [6] registered [ 0.393174] acpiphp: Slot [7] registered [ 0.395177] acpiphp: Slot [8] registered [ 0.397115] acpiphp: Slot [9] registered [ 0.398241] acpiphp: Slot [10] registered [ 0.400123] acpiphp: Slot [3] registered [ 0.401110] acpiphp: Slot [4] registered [ 0.402123] acpiphp: Slot [11] registered [ 0.403109] acpiphp: Slot [12] registered [ 0.404107] acpiphp: Slot [13] registered [ 0.406172] acpiphp: Slot [14] registered [ 0.407158] acpiphp: Slot [15] registered [ 0.408129] acpiphp: Slot [16] registered [ 0.409177] acpiphp: Slot [17] registered [ 0.411105] acpiphp: Slot [18] registered [ 0.412114] acpiphp: Slot [19] registered [ 0.413124] acpiphp: Slot [20] registered [ 0.415150] acpiphp: Slot [21] registered [ 0.416139] acpiphp: Slot [22] registered [ 0.417175] acpiphp: Slot [23] registered [ 0.419105] acpiphp: Slot [24] registered [ 0.420102] acpiphp: Slot [25] registered [ 0.421112] acpiphp: Slot [26] registered [ 0.422170] acpiphp: Slot [27] registered [ 0.423154] acpiphp: Slot [28] registered [ 0.425119] acpiphp: Slot [29] registered [ 0.426121] acpiphp: Slot [30] registered [ 0.427113] acpiphp: Slot [31] registered [ 0.428115] PCI host bridge to bus 0000:00 [ 0.430024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.432029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.434029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.436026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.438029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.440046] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.443106] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.447068] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.450311] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.459782] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.465551] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.467025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.469024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.471024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.474136] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.477826] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.480042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.483730] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.489017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.500998] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.507016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.514270] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.522020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.529021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.546022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.555304] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.565021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.574022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.593028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.603832] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.610020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.617025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.636021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.646160] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.655019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.663021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.681019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.696511] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.706023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.713017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.731038] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.743407] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.751017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.758016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.772013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.784147] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.788423] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.789309] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.792382] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.794267] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.800160] iommu: Default domain type: Passthrough [ 0.802686] SCSI subsystem initialized [ 0.804169] ACPI: bus type USB registered [ 0.807111] usbcore: registered new interface driver usbfs [ 0.809092] usbcore: registered new interface driver hub [ 0.811089] usbcore: registered new device driver usb [ 0.813215] pps_core: LinuxPPS API ver. 1 registered [ 0.816012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.820063] PTP clock support registered [ 0.822288] EDAC MC: Ver: 3.0.0 [ 0.823523] PCI: Using ACPI for IRQ routing [ 0.826172] NetLabel: Initializing [ 0.827015] NetLabel: domain hash size = 128 [ 0.828011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.829080] NetLabel: unlabeled traffic allowed by default [ 0.832186] vgaarb: loaded [ 0.833223] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.834016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.839796] clocksource: Switched to clocksource kvm-clock [ 0.945814] VFS: Disk quotas dquot_6.6.0 [ 0.947584] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.950218] *** VALIDATE ramfs *** [ 0.951520] *** VALIDATE hugetlbfs *** [ 0.953071] pnp: PnP ACPI init [ 0.956124] pnp: PnP ACPI: found 6 devices [ 0.973250] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.976903] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.979332] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.981661] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.984334] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.987037] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.989978] NET: Registered protocol family 2 [ 0.993258] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.999251] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.003311] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.009210] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.012806] TCP: Hash tables configured (established 65536 bind 65536) [ 1.016058] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.019801] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.023182] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.026599] NET: Registered protocol family 1 [ 1.031335] RPC: Registered named UNIX socket transport module. [ 1.033986] RPC: Registered udp transport module. [ 1.035898] RPC: Registered tcp transport module. [ 1.037891] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.041501] NET: Registered protocol family 44 [ 1.043437] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.045832] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.048404] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.050570] PCI: CLS 0 bytes, default 64 [ 1.052086] Unpacking initramfs... [ 2.449529] debug: unmapping init [mem 0xffff96703cc54000-0xffff96703ffbffff] [ 2.453938] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.456646] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.459905] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.989707] Initialise system trusted keyrings [ 2.991639] Key type blacklist registered [ 2.995538] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.006398] zbud: loaded [ 3.010181] *** VALIDATE nfs *** [ 3.011732] *** VALIDATE nfs4 *** [ 3.013886] pstore: using deflate compression [ 3.017818] Platform Keyring initialized [ 3.138786] NET: Registered protocol family 38 [ 3.140951] Key type asymmetric registered [ 3.142830] Asymmetric key parser 'x509' registered [ 3.144963] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.147471] io scheduler mq-deadline registered [ 3.149322] io scheduler kyber registered [ 3.151158] io scheduler bfq registered [ 3.153181] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.156722] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.159600] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.162415] ACPI: Power Button [PWRF] [ 3.168193] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.175801] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.195993] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.207303] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.232406] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.260035] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.287971] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.292979] Non-volatile memory driver v1.3 [ 3.294480] Linux agpgart interface v0.103 [ 3.329294] virtio_blk virtio1: [vda] 145784 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.332993] vda: detected capacity change from 0 to 74641408 [ 3.348779] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.352045] vdb: detected capacity change from 0 to 1073741824 [ 3.368430] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.371432] vdc: detected capacity change from 0 to 2621440000 [ 3.384699] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.387090] vdd: detected capacity change from 0 to 2621440000 [ 3.401171] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.403941] vde: detected capacity change from 0 to 4294967296 [ 3.419982] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.422380] vdf: detected capacity change from 0 to 4294967296 [ 3.429840] libphy: Fixed MDIO Bus: probed [ 3.434971] usbcore: registered new interface driver usbserial_generic [ 3.437828] usbserial: USB Serial support registered for generic [ 3.440312] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.445153] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.447299] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.449981] mousedev: PS/2 mouse device common for all mice [ 3.452670] rtc_cmos 00:05: RTC can wake from S4 [ 3.455476] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.457236] rtc_cmos 00:05: registered as rtc0 [ 3.461391] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.462342] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.467345] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.469228] intel_pstate: CPU model not supported [ 3.476216] hid: raw HID events driver (C) Jiri Kosina [ 3.479015] usbcore: registered new interface driver usbhid [ 3.481716] usbhid: USB HID core driver [ 3.483848] drop_monitor: Initializing network drop monitor service [ 3.487205] Initializing XFRM netlink socket [ 3.489268] NET: Registered protocol family 10 [ 3.492430] Segment Routing with IPv6 [ 3.493993] NET: Registered protocol family 17 [ 3.497043] mpls_gso: MPLS GSO support [ 3.504285] RAS: Correctable Errors collector initialized. [ 3.506916] AVX version of gcm_enc/dec engaged. [ 3.508584] AES CTR mode by8 optimization enabled [ 3.612076] sched_clock: Marking stable (3612041230, 0)->(4538123682, -926082452) [ 3.616329] registered taskstats version 1 [ 3.618540] Loading compiled-in X.509 certificates [ 3.620860] zswap: loaded using pool lzo/zbud [ 3.645029] Key type big_key registered [ 3.657487] Key type encrypted registered [ 3.659135] ima: No TPM chip found, activating TPM-bypass! [ 3.661259] ima: Allocated hash algorithm: sha1 [ 3.662812] ima: No architecture policies found [ 3.663856] evm: Initialising EVM extended attributes: [ 3.665650] evm: security.selinux [ 3.667333] evm: security.ima [ 3.668523] evm: security.capability [ 3.669849] evm: HMAC attrs: 0x1 [ 3.672791] rtc_cmos 00:05: setting system clock to 2026-07-18 19:53:30 UTC (1784404410) [ 3.680781] debug: unmapping init [mem 0xffffffff8ee03000-0xffffffff8effffff] [ 3.684273] debug: unmapping init [mem 0xffffffff8db82000-0xffffffff8de58fff] [ 3.691146] Write protecting the kernel read-only data: 28672k [ 3.694905] debug: unmapping init [mem 0xffffffff8c203000-0xffffffff8c3fffff] [ 3.698167] debug: unmapping init [mem 0xffffffff8cb14000-0xffffffff8cbfffff] [ 3.744416] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.753971] systemd[1]: Detected virtualization kvm. [ 3.756085] systemd[1]: Detected architecture x86-64. [ 3.758115] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.783904] systemd[1]: No hostname configured. [ 3.785073] systemd[1]: Set hostname to . [ 3.786943] random: systemd: uninitialized urandom read (16 bytes read) [ 3.788962] systemd[1]: Initializing machine ID from random generator. [ 3.813257] random: ln: uninitialized urandom read (6 bytes read) [ 3.904423] random: systemd: uninitialized urandom read (16 bytes read) [ 3.907758] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.913654] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.918899] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.472301] device-mapper: uevent: version 1.0.3 [ 4.474147] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.203765] virtio_net virtio0 ens2: renamed from eth0 [ 5.279816] scsi host0: ata_piix [ 5.284403] random: fast init done [ 5.286612] scsi host1: ata_piix [ 5.349937] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.352803] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.559914] dracut-initqueue[593]: RTNETLINK answers: File exists [ 10.176599] random: crng init done [ 10.178571] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.572968] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.845218] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.115860] SELinux: Disabled at runtime. [ 12.180917] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.189987] systemd[1]: Detected virtualization kvm. [ 12.191923] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.717872] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.721682] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.730346] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.735458] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.739570] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.746356] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.750513] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ 12.892207] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.242416] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.517697] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.574794] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.714262] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.739253] EDAC sbridge: Ver: 1.1.2 [ 15.310941] Key type dns_resolver registered [ 15.629155] NFS: Registering the id_resolver key type [ 15.631629] Key type id_resolver registered [ 15.633341] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg354-server login: [ 60.720306] hrtimer: interrupt took 4951692 ns [ 61.793824] libcfs: loading out-of-tree module taints kernel. [ 61.838278] Key type ._llcrypt registered [ 61.840537] Key type .llcrypt registered [ 61.932238] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 79.684890] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 80.983336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 80.995841] alg: No test for adler32 (adler32-zlib) [ 82.378231] Lustre: Lustre: Build Version: 2.17.54_174_g53e1600 [ 83.125278] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 84.896926] Key type lgssc registered [ 86.535865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.410139] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 140.554534] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 153.160405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 153.189225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 154.485905] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 154.539558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 154.645619] Lustre: lustre-MDT0000: new disk, initializing [ 154.731808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 154.761218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 159.441390] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.865896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 171.968888] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 172.001738] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 172.007903] Lustre: Skipped 1 previous similar message [ 172.094164] Lustre: lustre-MDT0001: new disk, initializing [ 172.173339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 172.215976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 172.226859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 176.688560] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.283309] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 189.582103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 189.826578] Lustre: lustre-OST0000: new disk, initializing [ 189.830286] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 189.837421] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 189.839557] Lustre: 8452:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 189.898261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.413444] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.066406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 195.079735] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 195.125311] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 205.619574] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 205.720675] Lustre: lustre-OST0001: new disk, initializing [ 205.727450] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 205.731958] Lustre: 9524:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 205.776969] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 206.704662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 206.716149] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 206.764395] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 211.074745] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.593920] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 226.671612] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 231.969520] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing check_logdir /tmp/testlogs/ [ 236.174788] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing yml_node [ 239.348541] Lustre: DEBUG MARKER: Client: 2.17.54.174 [ 241.659402] Lustre: DEBUG MARKER: MDS: 2.17.54.174 [ 244.036654] Lustre: DEBUG MARKER: OSS: 2.17.54.174 [ 245.502872] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Jul 18 15:57:30 EDT 2026 [ 261.695418] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 262.854442] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 278.496621] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 278.501755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 278.523816] Lustre: Skipped 2 previous similar messages [ 278.531797] Lustre: Skipped 1 previous similar message [ 283.327024] Lustre: server umount lustre-MDT0000 complete [ 286.172873] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784404692 with bad export cookie 14697751655412136385 [ 286.186526] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 286.187026] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 288.737364] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 288.737754] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.740110] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 288.740119] Lustre: Skipped 2 previous similar messages [ 288.750186] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 288.783129] Lustre: Skipped 2 previous similar messages [ 291.743858] LustreError: 6522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 291.745272] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 291.766562] Lustre: Skipped 2 previous similar messages [ 292.468276] Lustre: server umount lustre-MDT0001 complete [ 295.713277] Lustre: server umount lustre-OST0000 complete [ 299.120746] Lustre: server umount lustre-OST0001 complete [ 304.442283] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 311.659887] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 338.560644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 338.749848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 338.772440] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 338.857823] Lustre: lustre-MDT0000: new disk, initializing [ 338.914838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 338.925505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 342.035113] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 350.010183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 350.115604] Lustre: 15071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 350.157502] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 350.159916] Lustre: Skipped 1 previous similar message [ 350.213929] Lustre: lustre-MDT0001: new disk, initializing [ 350.273647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 350.303469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 350.310886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 352.955233] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 359.465077] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 363.888359] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 368.699908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 368.952067] Lustre: lustre-OST0000: new disk, initializing [ 368.957810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 368.964410] Lustre: 17037:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 369.041218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 370.872189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 370.881378] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 371.014341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 374.000435] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 381.943981] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 385.726175] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 385.897553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 389.569167] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 389.750458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 391.652801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 391.654795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 391.657051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 391.664780] LustreError: Skipped 1 previous similar message [ 391.674245] Lustre: Skipped 1 previous similar message [ 397.309456] Lustre: server umount lustre-OST0000 complete [ 401.376727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 401.379909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 401.399146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 401.403987] Lustre: Skipped 3 previous similar messages [ 414.176126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 414.346081] Lustre: server umount lustre-MDT0000 complete [ 416.939480] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784404823 with bad export cookie 14697751655412137918 [ 416.941340] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 416.952222] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 417.178020] Lustre: server umount lustre-MDT0001 complete [ 421.989635] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 16:00:27 (1784404827) [ 436.200237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 436.550872] LustreError: 19914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 436.566623] LustreError: 19914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 436.622230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 440.092520] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 441.825608] LustreError: 19915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 446.952043] LustreError: 19914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 447.134766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 451.088150] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 457.643987] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 462.522136] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 464.256481] Lustre: Failing over lustre-MDT0000 [ 464.485815] Lustre: server umount lustre-MDT0000 complete [ 466.914338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.918818] LustreError: 19915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 466.921781] Lustre: Skipped 2 previous similar messages [ 466.964803] LustreError: 19915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 481.494316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.618346] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 485.431252] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 486.883337] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 486.900905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 486.928065] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 493.462491] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 497.120485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 497.136157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 497.153863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 497.157615] Lustre: Skipped 5 previous similar messages [ 502.812594] Lustre: server umount lustre-MDT0000 complete [ 503.264737] LustreError: 20641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 503.275624] LustreError: 20641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 505.885193] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 505.886825] LustreError: 19895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784404912 with bad export cookie 14697751655412139332 [ 505.908497] LustreError: 19895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 506.170582] Lustre: server umount lustre-MDT0001 complete [ 513.015715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 513.457677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 513.464439] Lustre: Skipped 2 previous similar messages [ 516.723248] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 523.579603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 526.861360] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 532.530359] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 536.305378] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 538.112885] Lustre: Failing over lustre-MDT0000 [ 538.354266] Lustre: server umount lustre-MDT0000 complete [ 539.105356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.120397] Lustre: Skipped 1 previous similar message [ 539.123463] LustreError: 23261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.132030] LustreError: 23261:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 554.463563] Lustre: 3650:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784404945/real 1784404945] req@ffff9670874d8700 x1871083721629696/t0(0) o400->MGC192.168.203.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784404961 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 554.495551] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 555.368530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.714289] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcbf8e74231c008ee [ 564.726629] Lustre: MGC192.168.203.154@tcp: Connection restored to 0@lo (at 0@lo) [ 564.734668] Lustre: Skipped 1 previous similar message [ 568.104270] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 568.994554] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 568.997840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 569.012898] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 575.584107] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 579.552314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 579.559641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.572064] Lustre: Skipped 1 previous similar message [ 579.577576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 579.582152] Lustre: Skipped 3 previous similar messages [ 593.375167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 593.466951] Lustre: server umount lustre-MDT0000 complete [ 596.625471] LustreError: 23241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784405003 with bad export cookie 14697751655412140270 [ 596.630237] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 596.634323] LustreError: 23241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 596.875715] Lustre: server umount lustre-MDT0001 complete [ 603.557927] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 16:03:29 (1784405009) [ 616.590030] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 623.555857] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 653.395496] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 661.184855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 661.337341] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 661.354658] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 661.413669] Lustre: lustre-MDT0000: new disk, initializing [ 661.469459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 661.476195] Lustre: Skipped 2 previous similar messages [ 661.487729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 664.454254] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 672.200706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.309817] Lustre: 29632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 672.562455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 675.550624] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 681.963504] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 685.549273] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 690.034571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 690.226511] Lustre: 31598:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 691.490972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 691.515935] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 691.551813] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 691.566391] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 694.086734] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 700.355201] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 705.798767] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 722.412976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 722.426730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 722.444371] Lustre: Skipped 1 previous similar message [ 722.452985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 722.457960] Lustre: Skipped 6 previous similar messages [ 723.846236] Lustre: server umount lustre-OST0000 complete [ 732.879377] Lustre: server umount lustre-MDT0000 complete [ 735.556398] LustreError: 29624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784405142 with bad export cookie 14697751655412140683 [ 735.567804] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 735.568677] LustreError: 29624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 735.882431] Lustre: server umount lustre-MDT0001 complete [ 743.630290] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 745.515602] Key type lgssc unregistered [ 745.765868] LNet: 33571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.779352] LNetError: 33571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 745.801525] LNet: Removed LNI 192.168.203.154@tcp [ 746.501084] Key type .llcrypt unregistered [ 746.504623] Key type ._llcrypt unregistered [ 764.311903] Key type ._llcrypt registered [ 764.313919] Key type .llcrypt registered [ 764.400183] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 764.974861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 765.037299] alg: No test for adler32 (adler32-zlib) [ 765.995752] Lustre: Lustre: Build Version: 2.17.54_174_g53e1600 [ 766.190681] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 767.839176] Key type lgssc registered [ 768.636239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 773.861683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 773.877316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 775.218306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 778.286571] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 784.620902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 784.845215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 787.983045] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 793.339370] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 796.921953] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 801.364937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 801.617485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 805.544921] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 809.853133] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 810.675779] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 835.416378] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 16:07:21 (1784405241) [ 836.623354] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 837.899036] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 842.460702] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 16:07:28 (1784405248) [ 849.061737] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 16:07:34 (1784405254) [ 849.730353] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 856.101822] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 856.106853] Lustre: Skipped 2 previous similar messages [ 857.504631] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 16:07:43 (1784405263) [ 860.382978] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 860.387179] Lustre: Skipped 3 previous similar messages [ 870.938278] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 870.942263] Lustre: Skipped 6 previous similar messages [ 872.492244] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 16:07:58 (1784405278) [ 874.255751] Lustre: *** cfs_fail_loc=131b, val=2*** [ 876.324156] Lustre: *** cfs_fail_loc=131b, val=2*** [ 881.324446] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 16:08:07 (1784405287) [ 888.326575] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 888.330807] Lustre: Skipped 7 previous similar messages [ 889.260212] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 16:08:15 (1784405295) [ 896.993276] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 897.000431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.009660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 899.040758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 899.045207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 899.046889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 902.324930] Lustre: server umount lustre-OST0000 complete [ 907.747452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 907.757074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 907.771209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 907.774460] Lustre: Skipped 1 previous similar message [ 910.703375] Lustre: server umount lustre-MDT0000 complete [ 912.951315] LustreError: 34943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784405319 with bad export cookie 6831871649163107810 [ 912.951734] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 912.957442] LustreError: 34943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 913.161572] Lustre: server umount lustre-MDT0001 complete [ 919.439221] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 921.123982] Key type lgssc unregistered [ 921.301628] LNet: 42821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.315358] LNetError: 42821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.326354] LNet: Removed LNI 192.168.203.154@tcp [ 921.876158] Key type .llcrypt unregistered [ 921.878198] Key type ._llcrypt unregistered [ 932.985278] Key type ._llcrypt registered [ 932.987022] Key type .llcrypt registered [ 933.053542] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 942.379526] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 943.015846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 943.027741] alg: No test for adler32 (adler32-zlib) [ 943.973732] Lustre: Lustre: Build Version: 2.17.54_174_g53e1600 [ 944.126222] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 945.751201] Key type lgssc registered [ 946.476578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 968.848119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 968.870904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 970.015724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 970.036830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 970.085750] Lustre: lustre-MDT0000: new disk, initializing [ 970.122847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 970.139864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 972.786827] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 979.707537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 979.801278] Lustre: 45950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 979.856218] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 979.860906] Lustre: Skipped 1 previous similar message [ 979.909768] Lustre: lustre-MDT0001: new disk, initializing [ 979.972705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 980.002068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 980.010844] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 982.511115] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 988.525501] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 991.960776] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 996.580076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 996.757559] Lustre: lustre-OST0000: new disk, initializing [ 996.761328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 996.766727] Lustre: 47919:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 996.817665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1000.661762] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1002.532620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1002.538392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1002.603526] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1007.261385] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1010.256543] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1010.380854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1013.371641] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1013.493432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1017.827590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1017.831056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1017.833297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1017.840772] Lustre: Skipped 1 previous similar message [ 1020.800370] Lustre: server umount lustre-OST0000 complete [ 1026.019282] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1026.026498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1026.037906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1026.041750] Lustre: Skipped 1 previous similar message [ 1028.065447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1028.065523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1028.080874] Lustre: Skipped 1 previous similar message [ 1029.450313] Lustre: server umount lustre-MDT0000 complete [ 1031.786904] LustreError: 45943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784405438 with bad export cookie 9194635113444597918 [ 1031.787867] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1031.795579] LustreError: 45943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1032.053912] Lustre: server umount lustre-MDT0001 complete [ 1038.337296] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 16:10:44 (1784405444) [ 1042.584229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1042.975725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1045.652426] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1048.032691] LustreError: 50377:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1048.042446] LustreError: 50377:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1051.031228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1051.230107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1053.936836] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1058.285935] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1061.759867] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1066.157794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1066.406259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.821396] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1074.342602] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1081.431090] LustreError: 52905:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1082.038146] Lustre: Modifying parameter general.timeout=1 in log params [ 1083.112848] Lustre: Modifying parameter general.timeout=3 in log params [ 1083.116309] Lustre: Skipped 1 previous similar message [ 1085.392686] Lustre: Modifying parameter general.timeout=7 in log params [ 1085.395496] Lustre: Skipped 3 previous similar messages [ 1089.819695] Lustre: Modifying parameter general.timeout=15 in log params [ 1089.823556] Lustre: Skipped 7 previous similar messages [ 1098.263195] Lustre: Modifying parameter general.timeout=30 in log params [ 1098.266877] Lustre: Skipped 14 previous similar messages [ 1114.604225] Lustre: Modifying parameter general.timeout=59 in log params [ 1114.610983] Lustre: Skipped 28 previous similar messages [ 1147.081621] Lustre: Modifying parameter general.timeout=119 in log params [ 1147.085228] Lustre: Skipped 59 previous similar messages [ 1158.945311] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 16:12:44 (1784405564) [ 1160.731789] Lustre: 59819:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1161.827442] Lustre: 59917:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1164.830694] Lustre: 60212:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1164.838221] Lustre: 60212:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1180.822748] Lustre: 61690:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1180.828693] Lustre: 61690:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1180.843162] LustreError: 61690:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1183.222719] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 16:13:09 (1784405589) [ 1191.394970] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.396828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1191.411582] Lustre: Skipped 1 previous similar message [ 1191.416945] Lustre: Skipped 2 previous similar messages [ 1196.432932] Lustre: server umount lustre-MDT0000 complete [ 1196.513535] LustreError: 50372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1196.527228] LustreError: 50372:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1198.767983] LustreError: 50359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784405605 with bad export cookie 9194635113444598800 [ 1198.770256] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1198.781575] LustreError: 50359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1199.003663] Lustre: server umount lustre-MDT0001 complete [ 1211.711695] Lustre: server umount lustre-OST0000 complete [ 1227.122296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1227.264716] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1227.291906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1227.303348] Lustre: Skipped 33 previous similar messages [ 1227.449970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1229.696263] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1234.378040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1234.448647] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1234.466477] Lustre: 64220:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1236.954931] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1240.856456] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1243.796808] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1247.675581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1247.816198] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1251.403511] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1255.501446] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1256.491803] Lustre: 66784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1264.098444] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1264.098861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1264.104986] LustreError: Skipped 1 previous similar message [ 1264.111547] Lustre: Skipped 1 previous similar message [ 1264.112630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1264.119412] Lustre: Skipped 1 previous similar message [ 1266.570195] Lustre: server umount lustre-OST0000 complete [ 1274.337454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1274.344535] Lustre: Skipped 3 previous similar messages [ 1275.036625] Lustre: server umount lustre-MDT0000 complete [ 1277.093628] LustreError: 64210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784405683 with bad export cookie 9194635113444601467 [ 1277.098700] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1277.104028] LustreError: 64210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1277.278789] Lustre: server umount lustre-MDT0001 complete [ 1282.726772] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1284.139485] Key type lgssc unregistered [ 1284.296453] LNet: 67989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1284.300162] LNetError: 67989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1284.310821] LNet: Removed LNI 192.168.203.154@tcp [ 1284.807172] Key type .llcrypt unregistered [ 1284.810332] Key type ._llcrypt unregistered [ 1298.011743] Key type ._llcrypt registered [ 1298.013491] Key type .llcrypt registered [ 1298.070533] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1298.587186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1298.599112] alg: No test for adler32 (adler32-zlib) [ 1299.532770] Lustre: Lustre: Build Version: 2.17.54_174_g53e1600 [ 1299.691163] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1301.295395] Key type lgssc registered [ 1301.822903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1305.642205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1305.658107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1306.887599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1308.760599] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1312.750139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1313.040234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1315.024817] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1318.919685] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1321.579928] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1324.963301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1325.250974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1328.196214] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1332.159276] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1340.838531] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1347.392308] Lustre: server umount lustre-MDT0000 complete [ 1348.067330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1348.068657] LustreError: 71830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1348.073814] Lustre: Skipped 1 previous similar message [ 1348.080648] LustreError: 71830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1349.227205] LustreError: 69363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784405756 with bad export cookie 1626799873547995215 [ 1349.229826] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1349.237063] LustreError: 69363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1351.490734] Lustre: server umount lustre-MDT0001 complete [ 1363.706810] Lustre: server umount lustre-OST0000 complete [ 1377.259370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1377.409951] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1377.431767] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1377.599127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1379.660423] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1384.107564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1384.162661] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1384.181098] Lustre: 74246:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1384.425406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1386.615475] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1390.894925] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1394.183551] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1397.973174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1398.097113] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1398.344585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1401.368990] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1405.069521] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1405.977692] Lustre: 76820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1406.942450] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1414.119795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1414.120280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1414.120954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1414.128091] LustreError: Skipped 1 previous similar message [ 1414.147099] Lustre: Skipped 2 previous similar messages [ 1415.300065] Lustre: server umount lustre-OST0000 complete [ 1419.235806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1419.238032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1419.244326] Lustre: Skipped 1 previous similar message [ 1419.253047] Lustre: Skipped 2 previous similar messages [ 1423.623958] Lustre: server umount lustre-MDT0000 complete [ 1424.353172] LustreError: 74257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1424.362135] LustreError: 74257:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1425.421794] LustreError: 74239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784405832 with bad export cookie 1626799873547996027 [ 1425.423879] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1425.426643] LustreError: 74239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1425.590921] Lustre: server umount lustre-MDT0001 complete [ 1430.591985] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1432.032544] Key type lgssc unregistered [ 1432.209632] LNet: 78030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1432.214923] LNetError: 78030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1432.226530] LNet: Removed LNI 192.168.203.154@tcp [ 1432.689915] Key type .llcrypt unregistered [ 1432.693091] Key type ._llcrypt unregistered [ 1444.880988] Key type ._llcrypt registered [ 1444.882736] Key type .llcrypt registered [ 1444.954673] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1445.417134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1445.456074] alg: No test for adler32 (adler32-zlib) [ 1446.363136] Lustre: Lustre: Build Version: 2.17.54_174_g53e1600 [ 1446.486502] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1448.087178] Key type lgssc registered [ 1448.627960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1452.773966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1452.785282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1454.014349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1455.906838] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1460.154131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1460.473379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1462.415216] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1466.148423] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1468.843693] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1472.202558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1472.514821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1475.475492] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1479.126553] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1487.862781] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1488.743463] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1488.746499] Lustre: Skipped 1 previous similar message [ 1489.900411] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1489.902835] Lustre: Skipped 2 previous similar messages [ 1492.102206] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1492.105198] Lustre: Skipped 5 previous similar messages [ 1496.211402] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1496.214286] Lustre: Skipped 9 previous similar messages [ 1504.505690] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1504.508162] Lustre: Skipped 20 previous similar messages [ 1520.827522] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1520.831589] Lustre: Skipped 41 previous similar messages [ 1530.593724] Lustre: 87490:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1531.862549] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 16:18:57 (1784405937) [ 1532.570520] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1533.365194] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 16:18:59 (1784405939) [ 1534.945451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1534.949149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1534.955605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1536.995414] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1536.996142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1536.999319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1540.772795] Lustre: server umount lustre-OST0000 complete [ 1547.233268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1547.236860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1547.237214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1547.237219] Lustre: Skipped 1 previous similar message [ 1547.241606] Lustre: Skipped 1 previous similar message [ 1548.612541] Lustre: server umount lustre-MDT0000 complete [ 1550.434640] LustreError: 82945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784405957 with bad export cookie 8312047372892293280 [ 1550.440615] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1550.441467] LustreError: 82945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1550.584581] Lustre: server umount lustre-MDT0001 complete [ 1555.579506] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1556.952719] Key type lgssc unregistered [ 1557.125886] LNet: 88736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1557.130840] LNetError: 88736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1557.144895] LNet: Removed LNI 192.168.203.154@tcp [ 1557.569089] Key type .llcrypt unregistered [ 1557.570835] Key type ._llcrypt unregistered [ 1559.404508] Key type ._llcrypt registered [ 1559.406387] Key type .llcrypt registered [ 1559.455440] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1561.326448] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules [ 1561.797420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1561.842507] alg: No test for adler32 (adler32-zlib) [ 1562.723738] Lustre: Lustre: Build Version: 2.17.54_174_g53e1600 [ 1562.826077] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1564.424788] Key type lgssc registered [ 1564.937824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1565.436568] LustreError: 89904:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 1574.135147] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1578.183703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1586.589801] LustreError: 89904:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1586.604762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1587.851386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1589.677498] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1593.495967] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 16:19:59 (1784405999) [ 1595.771614] LustreError: 90960:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1595.777269] LustreError: 90960:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1595.783167] LustreError: 90960:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 1595.941705] Lustre: server umount lustre-MDT0000 complete [ 1601.853365] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1603.159957] Key type lgssc unregistered [ 1603.302660] LNet: 92452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1603.308687] LNetError: 92452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1603.318823] LNet: Removed LNI 192.168.203.154@tcp [ 1603.688101] Key type .llcrypt unregistered [ 1603.689988] Key type ._llcrypt unregistered [ 1612.434111] Key type ._llcrypt registered [ 1612.435450] Key type .llcrypt registered [ 1612.488436] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1612.878227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1612.885975] alg: No test for adler32 (adler32-zlib) [ 1613.817946] Lustre: Lustre: Build Version: 2.17.54_174_g53e1600 [ 1613.970284] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1615.583261] Key type lgssc registered [ 1616.043803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1619.370965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1619.381396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1620.612929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1622.503978] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1626.419940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1626.741333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1628.571069] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1631.631403] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1633.896439] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1636.512495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1636.768875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1638.939596] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1641.766654] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1673.412224] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 16:21:19 (1784406079) [ 1674.138784] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1674.906595] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 16:21:21 (1784406081) [ 1676.769237] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1676.770335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1676.775674] Lustre: Skipped 2 previous similar messages [ 1676.780443] Lustre: Skipped 2 previous similar messages [ 1681.890974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1681.894958] Lustre: Skipped 2 previous similar messages [ 1682.398182] Lustre: server umount lustre-MDT0000 complete [ 1683.888148] LustreError: 93626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784406090 with bad export cookie 5967220251621772437 [ 1683.890963] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1683.892491] LustreError: 93626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1684.024825] Lustre: server umount lustre-MDT0001 complete [ 1696.025959] Lustre: server umount lustre-OST0000 complete [ 1700.983083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1701.192760] LustreError: 97914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1701.202190] LustreError: 97914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1701.278934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1702.930218] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1706.467208] LustreError: 97915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1706.554496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1706.858202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1708.619444] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1711.901839] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1714.525096] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1720.070714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1720.168691] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1720.172360] LustreError: 97904:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1720.183152] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1720.187522] LustreError: 99994:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1720.191858] LustreError: 99994:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1720.194779] LustreError: 99994:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1720.235904] Lustre: server umount lustre-OST0000 complete [ 1720.237960] LustreError: 99994:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1724.305251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1724.426063] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1724.431330] LustreError: 97904:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1724.438416] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1724.444226] LustreError: 100457:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1724.448973] LustreError: 100457:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1724.452142] LustreError: 100457:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1724.494708] Lustre: server umount lustre-OST0000 complete [ 1724.497146] LustreError: 100457:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1729.123215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1729.224645] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1729.227503] Lustre: Found index 0 for lustre-OST0000, updating log [ 1729.231286] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1729.282949] Lustre: lustre-OST0000: new disk, initializing [ 1729.286236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1729.292057] Lustre: 100990:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1729.423829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1732.119728] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1733.908594] Lustre: Failing over lustre-OST0000 [ 1733.986693] Lustre: server umount lustre-OST0000 complete [ 1736.673624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1736.674268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1736.678092] Lustre: Skipped 1 previous similar message [ 1736.682072] Lustre: Skipped 1 previous similar message [ 1741.792702] LustreError: 97914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1741.799831] LustreError: 97914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1741.825798] Lustre: server umount lustre-MDT0000 complete [ 1743.543828] LustreError: 100014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784406150 with bad export cookie 5967220251621775167 [ 1743.546434] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1743.550967] LustreError: 100014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1754.138830] Lustre: server umount lustre-MDT0001 complete [ 1757.470178] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 16:22:43 (1784406163) [ 1764.102019] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1768.285033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1768.450435] LustreError: 103332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1768.503296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1770.102467] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1773.449659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1773.536125] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1775.362135] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1776.555957] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1779.173867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1779.436733] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1781.714947] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1783.523345] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1783.537204] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1783.537347] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1785.018598] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1785.056088] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1785.094880] Lustre: lustre-OST0001: new disk, initializing [ 1785.097278] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1785.101158] Lustre: 105647:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1787.533203] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1793.047533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1793.052127] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1793.104844] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1796.266550] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1806.349432] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1808.961473] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.54@tcp (stopping) [ 1809.378911] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1809.385030] Lustre: Skipped 2 previous similar messages [ 1814.108300] Lustre: server umount lustre-MDT0001 complete [ 1814.499428] LustreError: 103333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1814.507300] LustreError: 103333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1817.982036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1818.035544] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1818.038305] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1818.055511] Lustre: 103322:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1818.360662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1818.363923] Lustre: Skipped 3 previous similar messages [ 1819.880195] LustreError: 107367:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1819.884631] LustreError: 107367:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1819.889565] LustreError: 103325:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1819.927326] LustreError: 107375:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.154@tcp: cfg command failed: rc = -17 [ 1819.934220] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1819.934220] [ 1820.063959] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1823.718525] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1824.030245] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 16:23:50 (1784406230) [ 1830.379696] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1836.889658] Lustre: 109274:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1845.670588] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1846.883827] Lustre: 110407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1899.999669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1900.003647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1900.009935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1900.012881] Lustre: Skipped 4 previous similar messages [ 1902.364788] Lustre: server umount lustre-MDT0000 complete [ 1903.749700] LustreError: 105646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784406310 with bad export cookie 5967220251621776084 [ 1903.751269] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1903.755397] LustreError: 105646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1903.860678] Lustre: server umount lustre-MDT0001 complete [ 1915.848155] Lustre: server umount lustre-OST0000 complete [ 1927.521640] Lustre: server umount lustre-OST0001 complete [ 1929.707245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1933.463263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1936.833439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1940.529071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1947.281813] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1951.236373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1951.245183] Lustre: lustre-MDT0000: reset Object Index mappings [ 1951.446938] LustreError: 114347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1951.487338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1953.113677] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1956.324764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1956.336092] Lustre: lustre-MDT0001: reset Object Index mappings [ 1958.233565] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1959.340801] Lustre: 115494:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1961.804409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1964.223767] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1966.189248] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:227 to 0x280000400:257) [ 1967.623595] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1970.186368] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1973.227127] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 1976.295711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 1976.297469] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 1979.066846] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1980.531547] Lustre: 117372:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1999.229971] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 16:26:45 (1784406405) [ 2032.612417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2032.613528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2032.618430] Lustre: Skipped 6 previous similar messages [ 2032.623225] Lustre: Skipped 7 previous similar messages [ 2034.854762] Lustre: server umount lustre-MDT0000 complete [ 2036.250271] LustreError: 115096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784406443 with bad export cookie 5967220251621822900 [ 2036.253415] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2036.255157] LustreError: 115096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2037.728581] LustreError: 114343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2037.735654] LustreError: 114343:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2042.569512] Lustre: server umount lustre-MDT0001 complete [ 2050.120351] Lustre: server umount lustre-OST0000 complete [ 2057.777287] Lustre: server umount lustre-OST0001 complete [ 2060.483266] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 2064.550453] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2079.852497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2079.966761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2079.987058] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2080.030413] Lustre: lustre-MDT0000: new disk, initializing [ 2080.064924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2080.069084] Lustre: Skipped 3 previous similar messages [ 2080.078090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2081.816438] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2085.292415] Lustre: server umount lustre-MDT0000 complete [ 2089.462330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2089.619275] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2089.626428] Lustre: 121626:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2089.633840] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2091.392264] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2093.400541] Lustre: server umount lustre-MDT0000 complete [ 2096.978338] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 16:28:23 (1784406503) [ 2097.726868] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2098.488563] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 16:28:24 (1784406504) [ 2106.015274] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2110.675370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2112.741609] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2116.733716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2116.799313] Lustre: 123675:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2116.831886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2116.835515] Lustre: Skipped 1 previous similar message [ 2116.876577] Lustre: lustre-MDT0001: new disk, initializing [ 2116.934992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2116.939378] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2118.803419] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2121.434978] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2124.513908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2124.634664] Lustre: lustre-OST0000: new disk, initializing [ 2124.637322] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2124.641276] Lustre: 125253:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2125.019159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2125.023991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2125.063274] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2127.359848] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2132.491607] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2132.578108] Lustre: 126123:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2134.229982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2134.233792] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2134.283883] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2135.184899] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2140.555994] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2142.253584] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2300.876367] Lustre: 132501:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2300.881156] Lustre: 132501:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2300.885040] Lustre: 132501:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2300.889034] Lustre: 132501:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2300.893459] Lustre: 132501:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2300.897414] Lustre: 132501:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2402.671428] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:33:28 (1784406808) [ 2416.097525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2416.100348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2416.103970] Lustre: Skipped 5 previous similar messages [ 2416.108608] Lustre: Skipped 7 previous similar messages [ 2416.401437] Lustre: server umount lustre-MDT0000 complete [ 2417.830454] LustreError: 123667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784406824 with bad export cookie 5967220251621878935 [ 2417.834372] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2417.836944] LustreError: 123667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2421.215958] LustreError: 126131:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2421.223032] LustreError: 126131:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2424.094224] Lustre: server umount lustre-MDT0001 complete [ 2431.555848] Lustre: server umount lustre-OST0000 complete [ 2447.327185] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2447.394830] Lustre: server umount lustre-OST0001 complete [ 2449.633806] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 2453.181176] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2466.370388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2466.449152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2466.460975] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2466.463445] Lustre: Skipped 1 previous similar message [ 2466.500230] Lustre: lustre-MDT0000: new disk, initializing [ 2466.502290] Lustre: Skipped 1 previous similar message [ 2466.522537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2466.525363] Lustre: Skipped 5 previous similar messages [ 2466.531135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2467.995526] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2472.511029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2472.562071] Lustre: 135515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2472.667697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2474.062711] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2477.813245] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2479.712273] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2482.309133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2482.438945] Lustre: 137484:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2484.029571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2484.033947] Lustre: Skipped 1 previous similar message [ 2484.036598] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2484.078996] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2484.683729] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2488.981709] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2492.855848] Lustre: lustre-MDD0000: changelog on [ 2493.997145] Lustre: lustre-MDD0001: changelog on [ 3017.047812] Lustre: lustre-MDD0001: changelog off [ 3017.683934] Lustre: 138203:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 3018.008519] Lustre: lustre-MDD0000: changelog off [ 3021.793911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.794491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3021.799254] Lustre: Skipped 2 previous similar messages [ 3021.799982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3021.801846] LustreError: Skipped 1 previous similar message [ 3021.806740] Lustre: Skipped 5 previous similar messages [ 3025.686168] Lustre: server umount lustre-OST0000 complete [ 3041.247127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3041.347771] Lustre: server umount lustre-MDT0000 complete [ 3042.573462] LustreError: 137483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784407449 with bad export cookie 5967220251622684124 [ 3042.577280] LustreError: 137483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3042.578505] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3042.711832] Lustre: server umount lustre-MDT0001 complete [ 3046.407892] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3047.501370] Key type lgssc unregistered [ 3047.642466] LNet: 146086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3047.645678] LNetError: 146086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3047.654624] LNet: Removed LNI 192.168.203.154@tcp [ 3048.008140] Key type .llcrypt unregistered [ 3048.009974] Key type ._llcrypt unregistered [ 3055.086504] Key type ._llcrypt registered [ 3055.088217] Key type .llcrypt registered [ 3055.129600] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 3061.250015] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3061.535450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3061.569041] alg: No test for adler32 (adler32-zlib) [ 3062.418616] Lustre: Lustre: Build Version: 2.17.54_174_g53e1600 [ 3062.514901] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3064.095198] Key type lgssc registered [ 3064.439424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3075.354392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3075.363139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3076.444369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3076.453481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3076.484387] Lustre: lustre-MDT0000: new disk, initializing [ 3076.504244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3076.510975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3077.813941] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3081.840897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3081.886736] Lustre: 149259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3081.909204] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3081.911540] Lustre: Skipped 1 previous similar message [ 3081.946515] Lustre: lustre-MDT0001: new disk, initializing [ 3081.975673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3081.987492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3081.991944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3083.247877] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3086.789968] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3088.479963] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3090.625787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3090.719401] Lustre: lustre-OST0000: new disk, initializing [ 3090.721203] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3090.724251] Lustre: 151229:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3090.750293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3091.709861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3091.714141] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3091.745480] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3092.655990] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3096.203778] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3098.393966] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_module obdecho/obdecho [ 3099.004854] LustreError: 152231:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3099.008692] LustreError: 152231:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 3099.570577] LustreError: 152327:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 3102.688071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3102.690473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3102.694456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3107.296189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3107.297984] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3107.298589] Lustre: Skipped 1 previous similar message [ 3107.301902] Lustre: Skipped 1 previous similar message [ 3108.759679] Lustre: server umount lustre-MDT0000 complete [ 3109.860176] LustreError: 149253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784407516 with bad export cookie 10038147363921230670 [ 3109.863128] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3109.863638] LustreError: 149253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3109.958522] Lustre: server umount lustre-MDT0001 complete [ 3120.340647] Lustre: server umount lustre-OST0000 complete [ 3126.905021] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 3130.014867] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3141.035061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3141.110058] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3141.118809] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3141.152541] Lustre: lustre-MDT0000: new disk, initializing [ 3141.172899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3141.178858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3142.458632] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3146.493681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3146.535056] Lustre: 156031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3146.557600] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3146.559709] Lustre: Skipped 1 previous similar message [ 3146.596479] Lustre: lustre-MDT0001: new disk, initializing [ 3146.629514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3146.641286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3146.646091] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3147.918360] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3151.535309] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3153.343722] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3155.562777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3155.658708] Lustre: lustre-OST0000: new disk, initializing [ 3155.660891] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3155.663185] Lustre: 158002:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3155.690255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3157.633456] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3159.354995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3159.358597] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3159.396146] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3161.367700] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3162.855930] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3162.934304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3164.356386] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3164.430595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3169.760447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3169.760524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3169.766216] LustreError: Skipped 1 previous similar message [ 3169.772419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3169.775463] Lustre: Skipped 2 previous similar messages [ 3171.154205] Lustre: server umount lustre-OST0000 complete [ 3174.880693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3174.881317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3174.885936] Lustre: Skipped 2 previous similar messages [ 3178.648316] Lustre: server umount lustre-MDT0000 complete [ 3179.954248] LustreError: 156829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784407586 with bad export cookie 10038147363921231615 [ 3179.957015] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3179.959023] LustreError: 156829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3180.066733] Lustre: server umount lustre-MDT0001 complete [ 3181.330200] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 16:46:27 (1784407587) [ 3181.887562] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3182.502376] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 16:46:28 (1784407588) [ 3185.308321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3185.481253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3186.928798] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3190.017043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3191.588973] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3194.011267] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3195.825125] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3198.217088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3200.492220] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3202.955872] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3215.842783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3215.844148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3215.844700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3215.844708] Lustre: Skipped 1 previous similar message [ 3215.847092] Lustre: Skipped 2 previous similar messages [ 3221.408374] Lustre: server umount lustre-MDT0000 complete [ 3222.690097] LustreError: 160393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784407629 with bad export cookie 10038147363921232483 [ 3222.693384] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3222.695179] LustreError: 160393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3223.597754] LustreError: 160405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3223.606138] LustreError: 160405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3226.079776] LustreError: 160407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3226.079977] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3226.089148] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3226.090987] Lustre: Skipped 7 previous similar messages [ 3227.103696] LustreError: 162248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3228.858687] Lustre: server umount lustre-MDT0001 complete [ 3231.231385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3235.895251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3235.989254] LustreError: 164270:0:(llog.c:1655:llog_backup()) MGC192.168.203.154@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 3235.993239] Lustre: 164270:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.154@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 3252.704022] LustreError: 164275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3252.709826] LustreError: 164275:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3257.824202] LustreError: 164277:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3257.829170] LustreError: 164277:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3268.063972] LustreError: 164295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3268.070746] LustreError: 164295:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 3286.060491] LustreError: 164275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3286.068105] LustreError: 164275:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 3296.863239] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3296.865586] Lustre: 164270:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.154@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3320.876321] LustreError: 164295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3320.882676] LustreError: 164295:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 22 previous similar messages [ 3337.702300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3337.705758] Lustre: Skipped 2 previous similar messages [ 3339.091192] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3342.018827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3342.821947] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3342.825096] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8b4ea9d7d58c9b8b [ 3344.331950] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3346.744712] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3347.224856] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3347.940416] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3347.942397] Lustre: Skipped 1 previous similar message [ 3348.991329] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3351.563735] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3356.750169] Lustre: server umount lustre-MDT0000 complete [ 3358.008245] LustreError: 160436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784407764 with bad export cookie 10038147363921238923 [ 3358.012430] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3358.012510] LustreError: 160436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3358.176575] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3358.182800] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3358.185241] Lustre: Skipped 2 previous similar messages [ 3364.205650] Lustre: server umount lustre-MDT0001 complete [ 3365.582939] Lustre: server umount lustre-OST0000 complete [ 3372.611914] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 3376.157959] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3388.962467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3389.051025] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3389.063414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3389.108323] Lustre: lustre-MDT0000: new disk, initializing [ 3389.141865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3390.542866] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3394.981512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3395.029351] Lustre: 169342:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3395.144437] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3396.619316] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3400.521852] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3402.356114] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3404.775189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3404.898302] Lustre: 171308:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3404.931743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3404.934676] Lustre: Skipped 3 previous similar messages [ 3406.945489] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3410.746541] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3412.143174] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3412.471396] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3412.502938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3413.235889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3414.660797] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3414.743371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3417.569969] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3417.570922] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3417.572940] LustreError: Skipped 1 previous similar message [ 3417.577314] Lustre: Skipped 1 previous similar message [ 3417.578054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3417.583872] Lustre: Skipped 2 previous similar messages [ 3421.521449] Lustre: server umount lustre-OST0000 complete [ 3425.760623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3429.016232] Lustre: server umount lustre-MDT0000 complete [ 3430.333137] LustreError: 169333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784407837 with bad export cookie 10038147363921255723 [ 3430.335875] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3430.338117] LustreError: 169333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3430.458874] Lustre: server umount lustre-MDT0001 complete [ 3433.499094] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 16:50:39 (1784407839) [ 3435.902881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3436.060758] LustreError: 173719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3436.067920] LustreError: 173719:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 3437.690222] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3441.365684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3444.438232] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3450.090883] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3454.359698] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3459.623560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3464.608390] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3471.543481] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3480.677651] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 16:51:25 (1784407885) [ 3486.690154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3486.697840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3486.704359] Lustre: Skipped 3 previous similar messages [ 3486.707921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3486.710595] Lustre: Skipped 3 previous similar messages [ 3491.314372] Lustre: server umount lustre-OST0000 complete [ 3508.191252] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3508.281891] Lustre: server umount lustre-MDT0000 complete [ 3510.353334] LustreError: 173700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784407917 with bad export cookie 10038147363921256591 [ 3510.357853] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3510.359573] LustreError: 173700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3510.520374] Lustre: server umount lustre-MDT0001 complete [ 3516.559594] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3518.134642] Key type lgssc unregistered [ 3518.291709] LNet: 177444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3518.303268] LNetError: 177444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3518.317693] LNet: Removed LNI 192.168.203.154@tcp [ 3518.735196] Key type .llcrypt unregistered [ 3518.737176] Key type ._llcrypt unregistered [ 3528.166915] Key type ._llcrypt registered [ 3528.168352] Key type .llcrypt registered [ 3528.226271] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3528.570263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3528.579170] alg: No test for adler32 (adler32-zlib) [ 3529.474965] Lustre: Lustre: Build Version: 2.17.54_174_g53e1600 [ 3529.598687] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3531.207219] Key type lgssc registered [ 3531.738445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3535.756931] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3535.767718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3551.264449] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3556.383880] LustreError: 178666:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.154@tcp: failed processing log, type 4: rc = -110 [ 3561.503542] LustreError: 178666:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3561.512944] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 3561.517212] LustreError: MGC192.168.203.154@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 3561.522504] LustreError: 178666:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 3561.527829] LustreError: 178666:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 3561.532304] LustreError: 178666:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3561.571184] Lustre: server umount lustre-OST0000 complete [ 3561.573562] LustreError: 178666:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 3565.984892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3567.207907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3569.088292] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3573.496533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3573.704803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3575.863470] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3579.285534] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3582.132260] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3586.022280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3586.137741] LustreError: 180976:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3586.145539] Lustre: 180976:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.154@tcp: invalid local config log lustre-OST0000: rc = -22 [ 3586.237306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3589.250524] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3593.054592] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3594.208171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3594.211845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3594.218083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3597.792532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3597.793610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3598.818929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3598.822652] Lustre: Skipped 4 previous similar messages [ 3600.393991] Lustre: server umount lustre-MDT0000 complete [ 3602.152206] LustreError: 179145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784408008 with bad export cookie 9080410770829472709 [ 3602.155910] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3602.157739] LustreError: 179145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3602.911777] LustreError: 179892:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003c3f87f9 x1871087335923072/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3603.003576] Lustre: server umount lustre-MDT0001 complete [ 3614.282888] Lustre: server umount lustre-OST0000 complete [ 3619.588368] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 3625.277706] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3643.180282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3643.275453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3643.290494] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3643.332945] Lustre: lustre-MDT0000: new disk, initializing [ 3643.376795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3643.385365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3645.264939] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3651.219621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3651.303244] Lustre: 184494:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3651.351467] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3651.354854] Lustre: Skipped 1 previous similar message [ 3651.423829] Lustre: lustre-MDT0001: new disk, initializing [ 3651.491431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3651.523625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3651.527855] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3653.887630] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3659.415507] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3662.431698] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3666.160793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3666.342505] Lustre: lustre-OST0000: new disk, initializing [ 3666.346323] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3666.351562] Lustre: 186459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3666.400763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3669.198129] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3672.080705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3672.084964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3672.141172] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3673.481290] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3675.337794] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3675.439932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3677.193058] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3677.281254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3682.273692] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3682.274126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3682.277290] LustreError: Skipped 1 previous similar message [ 3682.278424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3682.283923] Lustre: Skipped 2 previous similar messages [ 3684.188842] Lustre: server umount lustre-OST0000 complete [ 3687.393506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3687.393931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3687.394660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3687.394667] Lustre: Skipped 1 previous similar message [ 3687.409090] Lustre: Skipped 1 previous similar message [ 3691.875944] Lustre: server umount lustre-MDT0000 complete [ 3692.512076] LustreError: 184506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3692.522651] LustreError: 184506:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3693.447470] LustreError: 184489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784408100 with bad export cookie 9080410770829473437 [ 3693.448686] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3693.454983] LustreError: 184489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3693.575899] Lustre: server umount lustre-MDT0001 complete [ 3697.044634] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 16:55:03 (1784408103) [ 3703.806609] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3705.049843] Key type lgssc unregistered [ 3705.194473] LNet: 189402:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3705.198037] LNetError: 189402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3705.205578] LNet: Removed LNI 192.168.203.154@tcp [ 3705.631267] Key type .llcrypt unregistered [ 3705.632695] Key type ._llcrypt unregistered [ 3714.080120] Key type ._llcrypt registered [ 3714.081742] Key type .llcrypt registered [ 3714.135551] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3714.484446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3714.503295] alg: No test for adler32 (adler32-zlib) [ 3715.365179] Lustre: Lustre: Build Version: 2.17.54_174_g53e1600 [ 3715.466497] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3717.063254] Key type lgssc registered [ 3717.526996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3720.535156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3720.544311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3721.712357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3723.302451] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3726.597429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3726.763321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3728.462831] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3731.281908] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3733.643024] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3736.679969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3736.796968] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3736.799325] LustreError: 192472:0:(llog.c:1696:llog_backup()) MGC192.168.203.154@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3736.804878] Lustre: 192472:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.154@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3736.864816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3739.244872] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3742.158448] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3743.057971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3743.059645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3743.063282] Lustre: Skipped 1 previous similar message [ 3748.321775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3748.327407] Lustre: Skipped 4 previous similar messages [ 3749.215448] Lustre: server umount lustre-MDT0000 complete [ 3750.847423] LustreError: 190594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784408157 with bad export cookie 552596695813559592 [ 3750.854226] LustreError: 190594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3750.856136] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3751.015203] Lustre: server umount lustre-MDT0001 complete [ 3763.032116] Lustre: server umount lustre-OST0000 complete [ 3766.291624] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 3770.062597] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3783.271775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3783.357679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3783.372287] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3783.412609] Lustre: lustre-MDT0000: new disk, initializing [ 3783.442657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3783.451175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3784.938529] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3789.524523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3789.582388] Lustre: 195990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3789.607961] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3789.609424] Lustre: Skipped 1 previous similar message [ 3789.611239] LustreError: 196772:0:(llog.c:1696:llog_backup()) MGC192.168.203.154@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3789.614854] LustreError: 196772:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 3789.618578] Lustre: 196772:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.154@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3789.631835] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3789.634088] Lustre: Skipped 1 previous similar message [ 3789.673675] Lustre: lustre-MDT0001: new disk, initializing [ 3789.718352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3789.735386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3789.741815] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3791.334456] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3795.224731] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3797.180499] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3799.477993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3799.565891] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3799.567599] Lustre: Skipped 1 previous similar message [ 3799.569419] LustreError: 197934:0:(llog.c:1696:llog_backup()) MGC192.168.203.154@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3799.572291] LustreError: 197934:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 3799.575100] Lustre: 197934:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.154@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3799.580050] Lustre: 197934:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3799.587742] Lustre: lustre-OST0000: new disk, initializing [ 3799.589560] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3799.591919] Lustre: 197958:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3799.625834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3801.631954] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3802.045584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3802.050245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3802.091075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3805.450859] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3806.870947] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3806.947910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3808.420289] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3808.502666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3812.320696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3812.323595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3812.327554] Lustre: Skipped 1 previous similar message [ 3812.329466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3812.331550] Lustre: Skipped 1 previous similar message [ 3815.317474] Lustre: server umount lustre-OST0000 complete [ 3817.441484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3817.442839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3817.447899] Lustre: Skipped 2 previous similar messages [ 3822.560360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3822.562657] Lustre: Skipped 2 previous similar messages [ 3822.748614] Lustre: server umount lustre-MDT0000 complete [ 3823.995130] LustreError: 195982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784408230 with bad export cookie 552596695813560278 [ 3823.997873] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3824.000031] LustreError: 195982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3824.114207] Lustre: server umount lustre-MDT0001 complete [ 3827.084040] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 16:57:13 (1784408233) [ 3832.703920] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3836.426543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3836.608968] LustreError: 201078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3836.616331] LustreError: 201078:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3836.625306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3838.158058] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3841.222937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3842.822132] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3843.793744] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3846.275539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3848.578141] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3851.763052] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3851.837862] Lustre: lustre-OST0001: new disk, initializing [ 3851.840244] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3851.845319] Lustre: 203385:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3854.080136] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3854.157867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3854.161789] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3854.203576] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3859.968136] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3866.987451] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3867.728865] Lustre: DEBUG MARKER: ADD OST3 [ 3869.625506] loop: module loaded [ 3869.662420] loop0: detected capacity change from 0 to 204800000 [ 3869.788952] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3869.794393] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3869.803333] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3872.224846] loop0: detected capacity change from 0 to 204800000 [ 3874.454130] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3874.519213] Lustre: lustre-OST0003: new disk, initializing [ 3874.521360] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3874.524507] Lustre: 205117:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3874.569403] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3874.572433] Lustre: Skipped 3 previous similar messages [ 3874.909764] Lustre: DEBUG MARKER: STOP OST3 [ 3875.648218] Lustre: Failing over lustre-OST0003 [ 3875.696711] Lustre: server umount lustre-OST0003 complete [ 3876.171985] LustreError: 202566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3876.939344] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3880.650731] Lustre: DEBUG MARKER: START OST3 again [ 3881.019293] LustreError: 202565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3881.025298] LustreError: 202565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3882.972887] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3883.059972] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 3884.275020] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3884.283745] Lustre: lustre-OST0003: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3885.054088] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3888.108505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3888.112930] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3888.122758] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 3898.664602] Lustre: Failing over lustre-OST0003 [ 3898.710045] Lustre: server umount lustre-OST0003 complete [ 3903.463061] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3903.463193] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3903.466025] LustreError: Skipped 2 previous similar messages [ 3903.466615] LustreError: 202567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3903.471625] Lustre: Skipped 1 previous similar message [ 3903.480487] LustreError: 202567:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3908.582199] LustreError: 204174:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3908.588422] LustreError: 204174:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3913.184312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3913.187212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3913.192207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3917.345935] Lustre: server umount lustre-MDT0000 complete [ 3918.622427] LustreError: 201058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784408325 with bad export cookie 552596695813561146 [ 3918.627051] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3918.627829] LustreError: 201058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3918.816138] LustreError: 202585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3918.822243] LustreError: 202585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3924.911730] Lustre: server umount lustre-MDT0001 complete [ 3926.219457] Lustre: server umount lustre-OST0000 complete [ 3927.495264] Lustre: server umount lustre-OST0001 complete [ 3929.525782] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 3932.070883] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3944.893509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3944.978666] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3944.992592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3945.027086] Lustre: lustre-MDT0000: new disk, initializing [ 3945.046613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3945.048632] Lustre: Skipped 1 previous similar message [ 3945.052941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3946.467320] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3950.640816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3950.682717] Lustre: 210530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3950.799290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3952.171702] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3955.829278] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3957.616458] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3959.919845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3960.034384] Lustre: 212495:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3961.729524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3961.732896] Lustre: Skipped 1 previous similar message [ 3961.735202] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3961.784803] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3962.086062] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3965.916288] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3967.510472] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3967.615579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3969.221603] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3969.318826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3972.065831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3972.068196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3972.069605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3972.074053] Lustre: Skipped 6 previous similar messages [ 3972.075996] Lustre: Skipped 7 previous similar messages [ 3984.351107] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3984.411305] Lustre: server umount lustre-OST0000 complete [ 3986.911912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3986.914369] LustreError: Skipped 1 previous similar message [ 4000.223116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4000.285908] Lustre: server umount lustre-MDT0000 complete [ 4001.522388] LustreError: 212494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784408408 with bad export cookie 552596695813563386 [ 4001.525353] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4001.526501] LustreError: 212494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4001.645154] Lustre: server umount lustre-MDT0001 complete [ 4002.768743] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 17:00:09 (1784408409) [ 4007.917328] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 4011.168720] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4023.386933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4023.473134] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4023.484560] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4023.486434] Lustre: Skipped 3 previous similar messages [ 4023.522500] Lustre: lustre-MDT0000: new disk, initializing [ 4023.524305] Lustre: Skipped 2 previous similar messages [ 4023.549677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4024.906075] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4028.915638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4028.953021] Lustre: 217099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4029.054739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4030.353862] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4034.002760] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4035.870438] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4038.097714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4038.207042] Lustre: 219066:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4039.703324] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4040.268331] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4044.007462] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4045.569202] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4045.658945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4047.126502] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4047.223169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4049.890561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4049.890980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4049.892969] LustreError: Skipped 1 previous similar message [ 4049.896681] Lustre: Skipped 2 previous similar messages [ 4049.897428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4049.901142] Lustre: Skipped 12 previous similar messages [ 4062.175115] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4062.224825] Lustre: server umount lustre-OST0000 complete [ 4069.533808] Lustre: server umount lustre-MDT0000 complete [ 4070.368027] LustreError: 217111:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4070.372794] LustreError: 217111:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4070.752971] LustreError: 217091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784408477 with bad export cookie 552596695813564268 [ 4070.755942] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4070.757217] LustreError: 217091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4070.875047] Lustre: server umount lustre-MDT0001 complete [ 4073.731944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4073.907406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4073.910344] Lustre: Skipped 5 previous similar messages [ 4075.305888] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4078.324867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4079.985523] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4082.457881] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4084.292532] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4086.675487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4088.599940] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4091.131616] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4109.283279] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4109.283432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4109.286512] LustreError: Skipped 2 previous similar messages [ 4109.290232] Lustre: Skipped 3 previous similar messages [ 4113.994595] Lustre: server umount lustre-OST0000 complete [ 4119.522271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4119.524208] Lustre: Skipped 9 previous similar messages [ 4121.358427] Lustre: server umount lustre-MDT0000 complete [ 4122.571233] LustreError: 221272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784408529 with bad export cookie 552596695813565150 [ 4122.573682] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4122.574834] LustreError: 221272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4122.693023] Lustre: server umount lustre-MDT0001 complete [ 4224.593953] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 4225.529749] Key type lgssc unregistered [ 4225.657274] LNet: 224972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4225.659839] LNetError: 224972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4225.668513] LNet: Removed LNI 192.168.203.154@tcp [ 4225.987103] Key type .llcrypt unregistered [ 4225.988613] Key type ._llcrypt unregistered [ 4233.833955] Key type ._llcrypt registered [ 4233.835694] Key type .llcrypt registered [ 4233.871932] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 4240.197634] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4240.588766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4240.599039] alg: No test for adler32 (adler32-zlib) [ 4241.451994] Lustre: Lustre: Build Version: 2.17.54_174_g53e1600 [ 4241.541814] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 4243.119169] Key type lgssc registered [ 4243.475686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4254.775451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4254.782274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4255.862823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4255.873663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4255.912206] Lustre: lustre-MDT0000: new disk, initializing [ 4255.931652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4255.937059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4257.213500] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4261.199571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4261.246301] Lustre: 228296:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4261.273793] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4261.275569] Lustre: Skipped 1 previous similar message [ 4261.308764] Lustre: lustre-MDT0001: new disk, initializing [ 4261.344376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4261.358735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4261.363782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4262.741122] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4266.439612] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4268.284506] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4270.590194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4270.684717] Lustre: lustre-OST0000: new disk, initializing [ 4270.686318] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4270.688507] Lustre: 230262:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4270.716464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4272.643288] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4272.698206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4272.703162] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4272.739347] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4276.525452] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4278.053773] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4278.148125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4279.605874] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4279.684863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4282.385840] Lustre: server umount lustre-OST0000 complete [ 4286.944196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4286.947277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4286.953254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4287.968246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4287.968580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4293.088634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4293.091018] Lustre: Skipped 2 previous similar messages [ 4294.136762] Lustre: server umount lustre-MDT0000 complete [ 4295.393278] LustreError: 228287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784408702 with bad export cookie 18305459337152442694 [ 4295.397055] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4295.400342] LustreError: 228287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4305.943534] Lustre: server umount lustre-MDT0001 complete [ 4311.638572] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4314.861918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4315.023124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4316.317569] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4319.006155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4319.125261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4320.466380] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4321.431869] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4323.493721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4325.446526] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4328.194098] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4328.260110] Lustre: lustre-OST0001: new disk, initializing [ 4328.262367] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4328.265466] Lustre: 235497:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4328.300978] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4328.303615] Lustre: Skipped 1 previous similar message [ 4330.190378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4330.194708] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4330.200585] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4330.229427] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4335.702851] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4343.463769] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4346.869126] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 17:05:53 (1784408753) [ 4349.920190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4349.923174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4349.928393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4353.623410] Lustre: server umount lustre-MDT0000 complete [ 4354.734170] LustreError: 233926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784408761 with bad export cookie 18305459337152443562 [ 4354.736792] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4354.738122] LustreError: 233926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4354.859861] Lustre: server umount lustre-MDT0001 complete [ 4365.247545] Lustre: server umount lustre-OST0000 complete [ 4376.677333] Lustre: server umount lustre-OST0001 complete [ 4378.765340] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 4381.821364] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4393.056858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4393.131528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4393.141599] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4393.173125] Lustre: lustre-MDT0000: new disk, initializing [ 4393.189952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4393.195168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4394.474035] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4398.533873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4398.581057] Lustre: 239434:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4398.701729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4400.009407] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4403.593239] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4405.305434] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4407.415919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4407.504973] Lustre: lustre-OST0000: new disk, initializing [ 4407.506153] Lustre: Skipped 1 previous similar message [ 4407.507672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4407.509452] Lustre: Skipped 2 previous similar messages [ 4407.511698] Lustre: 241402:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4409.077173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4409.080572] Lustre: Skipped 1 previous similar message [ 4409.082815] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4409.111059] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4409.323159] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4412.882694] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4414.308551] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4414.392530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4415.808734] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4415.881662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4419.554595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4419.554666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4419.558656] Lustre: Skipped 4 previous similar messages [ 4419.558814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4419.558818] Lustre: Skipped 4 previous similar messages [ 4422.673974] Lustre: server umount lustre-OST0000 complete [ 4424.159897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4424.162558] LustreError: Skipped 1 previous similar message [ 4424.163768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4429.792075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4429.794791] Lustre: Skipped 4 previous similar messages [ 4430.043239] Lustre: server umount lustre-MDT0000 complete [ 4431.337294] LustreError: 239426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784408838 with bad export cookie 18305459337152445655 [ 4431.341411] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4431.342290] LustreError: 239426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4431.464244] Lustre: server umount lustre-MDT0001 complete [ 4434.336404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4434.495025] LustreError: 243614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4434.501510] LustreError: 243614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4434.517517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4434.520030] Lustre: Skipped 2 previous similar messages [ 4435.923407] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4438.792654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4440.285849] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4442.818865] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4444.750798] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4447.033446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4449.176360] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4451.706790] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4596.703810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4596.706129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4596.709813] Lustre: Skipped 1 previous similar message [ 4596.711606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4601.227527] Lustre: server umount lustre-OST0000 complete [ 4608.531934] Lustre: server umount lustre-MDT0000 complete [ 4609.732333] LustreError: 243594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784409016 with bad export cookie 18305459337152446537 [ 4609.735688] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4609.737009] LustreError: 243594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4609.843507] Lustre: server umount lustre-MDT0001 complete [ 4717.112233] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 4718.234166] Key type lgssc unregistered [ 4718.357362] LNet: 247217:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4718.360100] LNetError: 247217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4718.370264] LNet: Removed LNI 192.168.203.154@tcp [ 4718.720125] Key type .llcrypt unregistered [ 4718.721301] Key type ._llcrypt unregistered [ 4726.940227] Key type ._llcrypt registered [ 4726.941307] Key type .llcrypt registered [ 4726.980335] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 4733.423719] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4733.777430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4733.824099] alg: No test for adler32 (adler32-zlib) [ 4734.677288] Lustre: Lustre: Build Version: 2.17.54_174_g53e1600 [ 4734.764890] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 4736.351122] Key type lgssc registered [ 4736.722271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4751.468422] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4754.858475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4754.863642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4755.945753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4755.955880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4755.991963] Lustre: lustre-MDT0000: new disk, initializing [ 4756.013575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4756.019658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4757.325717] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4761.576272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4761.623282] Lustre: 251259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4761.653500] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4761.655560] Lustre: Skipped 1 previous similar message [ 4761.693070] Lustre: lustre-MDT0001: new disk, initializing [ 4761.728336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4761.740600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4761.743892] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4763.192428] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4765.500658] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4767.691464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4767.794897] Lustre: lustre-OST0000: new disk, initializing [ 4767.796536] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4767.799782] Lustre: 252888:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4767.830513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4769.837931] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4773.888792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4773.894929] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4773.939340] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4774.158573] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4774.247358] Lustre: lustre-OST0001: new disk, initializing [ 4774.250265] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4774.255430] Lustre: 253761:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4774.301889] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4776.321120] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4781.879824] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4782.080591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4782.083571] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4782.109996] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4787.504854] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4792.289684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4792.291808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4792.295167] Lustre: Skipped 1 previous similar message [ 4792.298806] Lustre: Skipped 3 previous similar messages [ 4796.399402] Lustre: server umount lustre-MDT0000 complete [ 4797.598303] LustreError: 252053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784409204 with bad export cookie 14225966627456117664 [ 4797.601652] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4797.605705] LustreError: 252053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4797.725796] Lustre: server umount lustre-MDT0001 complete [ 4808.706465] Lustre: server umount lustre-OST0000 complete [ 4820.123969] Lustre: server umount lustre-OST0001 complete [ 4824.313365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4827.308876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4829.858579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4832.727948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4838.709310] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4841.961506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4841.969517] Lustre: lustre-MDT0000: reset Object Index mappings [ 4842.104756] LustreError: 258765:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4842.111852] LustreError: 258765:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4842.119508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4843.330872] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4846.055034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4846.062760] Lustre: lustre-MDT0001: reset Object Index mappings [ 4847.513193] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4848.434180] Lustre: 259904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4850.542407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4850.684466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4850.687391] Lustre: Skipped 1 previous similar message [ 4851.684077] LustreError: 260259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4851.686775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4852.563918] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4855.204708] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4856.303397] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4857.142333] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4860.178161] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4861.271398] Lustre: 261772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4866.530229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4866.532831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4866.534547] Lustre: Skipped 5 previous similar messages [ 4866.537765] Lustre: Skipped 3 previous similar messages [ 4868.440853] Lustre: server umount lustre-MDT0000 complete [ 4869.584332] LustreError: 258745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784409276 with bad export cookie 14225966627456124440 [ 4869.587073] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4869.587881] LustreError: 258745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4869.703526] Lustre: server umount lustre-MDT0001 complete [ 4880.833426] Lustre: server umount lustre-OST0000 complete [ 4891.296846] Lustre: server umount lustre-OST0001 complete [ 4897.281932] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 17:15:03 (1784409303) [ 4902.582969] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 4906.055420] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4917.500580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4917.595779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4917.605540] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4917.639964] Lustre: lustre-MDT0000: new disk, initializing [ 4917.658380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4917.660202] Lustre: Skipped 1 previous similar message [ 4917.664131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4918.966618] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4922.996019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4923.036072] Lustre: 266013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4923.146314] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4924.404677] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4927.847751] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4929.457530] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4931.781454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4931.902952] Lustre: lustre-OST0000: new disk, initializing [ 4931.904684] Lustre: Skipped 1 previous similar message [ 4931.906472] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4931.908639] Lustre: Skipped 2 previous similar messages [ 4931.910634] Lustre: 267983:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4933.311149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4933.315031] Lustre: Skipped 1 previous similar message [ 4933.317789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4933.359469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4933.907989] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4937.759927] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4939.255994] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4939.333928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4940.729731] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4940.799618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4943.840977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4943.843692] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4943.845226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4943.848943] Lustre: Skipped 1 previous similar message [ 4943.852754] Lustre: Skipped 1 previous similar message [ 4947.544086] Lustre: server umount lustre-OST0000 complete [ 4948.960596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4948.960744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4948.961440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4948.963210] LustreError: Skipped 1 previous similar message [ 4948.969616] Lustre: Skipped 1 previous similar message [ 4954.080140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4954.082967] Lustre: Skipped 2 previous similar messages [ 4963.295281] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4963.296689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4963.300830] Lustre: Skipped 3 previous similar messages [ 4963.359884] Lustre: server umount lustre-MDT0000 complete [ 4964.634165] LustreError: 266006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784409371 with bad export cookie 14225966627456126512 [ 4964.634976] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4964.638461] LustreError: 266006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4964.765754] Lustre: server umount lustre-MDT0001 complete [ 4970.845483] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4974.343371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4974.473313] LustreError: 270908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4974.487358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4974.489362] Lustre: Skipped 2 previous similar messages [ 4975.836616] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4978.666363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4980.149403] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4981.135253] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4983.288279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4985.270964] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4988.230707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4988.306214] Lustre: lustre-OST0001: new disk, initializing [ 4988.307766] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4988.309867] Lustre: 273213:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4989.962602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4989.966872] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4990.010557] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4990.472960] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4994.962684] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4996.247928] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4998.000392] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4998.032460] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 5029.856145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5029.859224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5029.864617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5032.411130] Lustre: server umount lustre-MDT0000 complete [ 5033.649648] LustreError: 270889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784409440 with bad export cookie 14225966627456127394 [ 5033.654582] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5033.655106] LustreError: 270889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5033.778627] Lustre: server umount lustre-MDT0001 complete [ 5045.186054] Lustre: server umount lustre-OST0000 complete [ 5056.672353] Lustre: server umount lustre-OST0001 complete [ 5061.687709] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5065.529392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5065.682394] LustreError: 276292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5065.699834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5065.702309] Lustre: Skipped 3 previous similar messages [ 5067.221979] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5070.087920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5071.606162] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5072.554354] Lustre: 277435:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5074.619903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5075.747232] LustreError: 277788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5075.750392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 5076.537700] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5079.291042] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5080.424121] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 5081.209035] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5084.211165] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5085.351380] Lustre: 279299:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5087.121991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 5087.123745] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 5089.914833] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 17:18:16 (1784409496) [ 5106.147256] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5106.148657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5106.150886] Lustre: Skipped 5 previous similar messages [ 5106.151164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5106.152925] Lustre: Skipped 6 previous similar messages [ 5107.496797] Lustre: server umount lustre-MDT0000 complete [ 5108.631300] LustreError: 276274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784409515 with bad export cookie 14225966627456134163 [ 5108.634717] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5108.636994] LustreError: 276274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5111.264144] LustreError: 276288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5111.268532] LustreError: 276288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5114.806144] Lustre: server umount lustre-MDT0001 complete [ 5122.174655] Lustre: server umount lustre-OST0000 complete [ 5129.628216] Lustre: server umount lustre-OST0001 complete [ 5131.923178] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 5135.575332] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5149.803876] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5152.902598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5152.988547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5152.998627] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5153.035323] Lustre: lustre-MDT0000: new disk, initializing [ 5153.061404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5154.313351] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5158.410446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5158.455604] Lustre: 283533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5158.560171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5159.976959] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5164.485804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5164.593984] Lustre: 285166:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5165.862869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5166.614836] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5170.803434] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5172.516583] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5172.814515] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5177.345910] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5182.853881] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 17:19:49 (1784409589) [ 5184.479928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5184.482298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5184.486083] Lustre: Skipped 3 previous similar messages [ 5184.488068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5184.489952] Lustre: Skipped 4 previous similar messages [ 5189.726405] Lustre: server umount lustre-MDT0000 complete [ 5190.931197] LustreError: 283526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784409597 with bad export cookie 14225966627456141583 [ 5190.935120] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5190.937450] LustreError: 283526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5191.049991] Lustre: server umount lustre-MDT0001 complete [ 5202.441035] Lustre: server umount lustre-OST0000 complete [ 5213.993045] Lustre: server umount lustre-OST0001 complete [ 5216.927353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5217.100540] LustreError: 288282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5217.105202] LustreError: 288282:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5217.134403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5217.138126] Lustre: Skipped 7 previous similar messages [ 5218.524361] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5221.436794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5222.998678] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5225.377903] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5227.204180] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5229.445783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5231.428442] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5233.755186] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5249.583550] LustreError: 290110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5249.588633] LustreError: 290110:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 5250.502535] loop1: detected capacity change from 0 to 204800000 [ 5250.587318] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5250.590831] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5250.594608] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5252.285750] loop1: detected capacity change from 0 to 204800000 [ 5254.257834] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5254.291899] LustreError: 288271:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs16941: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 5254.296851] LustreError: 288271:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs16941 [ 5254.299048] LustreError: 288271:0:(mgs_handler.c:517:mgs_target_reg()) fs16941-MDT0001: Error checking target index: rc = -13 [ 5254.302578] LustreError: fs16941-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5254.305620] LustreError: 292013:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 5254.308034] LustreError: 292013:0:(tgt_mount.c:2129:server_put_super()) no obd fs16941-MDT0001 [ 5254.310162] LustreError: 292013:0:(tgt_mount.c:116:server_deregister_mount()) fs16941-MDT0001 not registered [ 5254.320987] Lustre: server umount fs16941-MDT0001 complete [ 5254.322735] LustreError: 292013:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5257.124049] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5257.152993] Lustre: Setting parameter fs16941-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs16941-MDT0001 [ 5257.156186] Lustre: Skipped 2 previous similar messages [ 5257.169092] Lustre: srv-fs16941-MDT0001: No data found on store. Initialize space. [ 5257.170775] Lustre: Skipped 4 previous similar messages [ 5257.204521] Lustre: fs16941-MDT0001: new disk, initializing [ 5257.206063] Lustre: Skipped 3 previous similar messages [ 5257.234972] Lustre: srv-fs16941-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 5257.237824] Lustre: 292480:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16941-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5257.241146] LustreError: 292480:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16941-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5258.474644] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5261.262933] Lustre: server umount fs16941-MDT0001 complete [ 5265.294449] loop1: detected capacity change from 0 to 204800000 [ 5267.083145] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5267.175521] Lustre: 294044:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16941-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5267.178243] LustreError: 294044:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16941-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5268.354154] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5269.852101] Lustre: server umount fs16941-MDT0001 complete [ 5273.055843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5273.058730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5273.062395] Lustre: Skipped 3 previous similar messages [ 5287.391116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5287.458143] Lustre: server umount lustre-MDT0000 complete [ 5290.274803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5290.314536] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5291.770997] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5294.282671] loop1: detected capacity change from 0 to 204800000 [ 5295.592112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5296.122766] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5296.233466] Lustre: 296489:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16941-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5296.236492] LustreError: 296489:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16941-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5297.514134] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5299.095107] Lustre: server umount fs16941-MDT0001 complete [ 5305.827317] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5316.069483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5316.071088] LustreError: 290110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5316.073912] Lustre: Skipped 19 previous similar messages [ 5316.091998] LustreError: 290110:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [ 5320.159093] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5320.212516] Lustre: server umount lustre-OST0000 complete [ 5327.706251] Lustre: server umount lustre-MDT0000 complete [ 5328.963331] LustreError: 288264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784409735 with bad export cookie 14225966627456144257 [ 5328.965653] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5328.966286] LustreError: 288264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5329.092792] Lustre: server umount lustre-MDT0001 complete [ 5332.965753] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 5334.066292] Key type lgssc unregistered [ 5334.203534] LNet: 298829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5334.206519] LNetError: 298829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5334.218450] LNet: Removed LNI 192.168.203.154@tcp [ 5334.558134] Key type .llcrypt unregistered [ 5334.559271] Key type ._llcrypt unregistered [ 5342.130693] Key type ._llcrypt registered [ 5342.131764] Key type .llcrypt registered [ 5342.169088] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5342.579098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5342.628136] alg: No test for adler32 (adler32-zlib) [ 5343.492017] Lustre: Lustre: Build Version: 2.17.54_174_g53e1600 [ 5343.586203] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 5345.175132] Key type lgssc registered [ 5345.562266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5348.075346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5348.081343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5349.233918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5350.568322] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5353.412740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5353.565301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5355.108936] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5357.603215] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5359.521591] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5361.867250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5362.019183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5364.059521] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5364.069764] LustreError: 301904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5364.079622] LustreError: 301904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5366.575079] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5369.317353] LustreError: 301904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5374.435790] LustreError: 301906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5374.444214] LustreError: 301906:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5377.075445] LustreError: 301906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5382.190466] LustreError: 301905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5382.196516] LustreError: 301905:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5383.115465] loop1: detected capacity change from 0 to 204800000 [ 5383.234972] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5383.238654] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5383.242545] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5384.981520] loop1: detected capacity change from 0 to 204800000 [ 5386.880870] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5386.926799] Lustre: fs26924-OST0000: new disk, initializing [ 5386.928450] Lustre: srv-fs26924-OST0000: No data found on store. Initialize space. [ 5386.930738] Lustre: 303830:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5386.951753] Lustre: fs26924-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5388.922581] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5392.347823] Lustre: server umount fs26924-OST0000 complete [ 5392.430161] LustreError: 302585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5392.437607] LustreError: 302585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5396.577421] loop1: detected capacity change from 0 to 204800000 [ 5398.420455] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5398.490711] Lustre: fs26924-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5400.328855] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5403.155148] loop2: detected capacity change from 0 to 204800000 [ 5403.236899] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5403.240768] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5403.244339] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5404.950164] loop2: detected capacity change from 0 to 204800000 [ 5406.888181] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5406.919198] LustreError: fs26924-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 5406.924077] LustreError: fs26924-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5406.929608] LustreError: 306865:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 5406.933949] LustreError: 306865:0:(tgt_mount.c:2129:server_put_super()) no obd fs26924-OST0001 [ 5406.937842] LustreError: 306865:0:(tgt_mount.c:116:server_deregister_mount()) fs26924-OST0001 not registered [ 5406.953064] Lustre: server umount fs26924-OST0001 complete [ 5406.955558] LustreError: 306865:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5410.277912] LustreError: 302585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5410.283148] LustreError: 302585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 5410.515969] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5410.569824] Lustre: fs26924-OST0001: new disk, initializing [ 5410.572433] Lustre: srv-fs26924-OST0001: No data found on store. Initialize space. [ 5410.576493] Lustre: 307418:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5410.603791] Lustre: fs26924-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5412.636748] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5416.190510] Lustre: server umount fs26924-OST0000 complete [ 5419.324957] Lustre: server umount fs26924-OST0001 complete [ 5423.402117] loop1: detected capacity change from 0 to 204800000 [ 5425.355480] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5427.457936] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5429.747329] loop2: detected capacity change from 0 to 204800000 [ 5431.696345] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5431.763724] Lustre: fs26924-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5431.766470] Lustre: Skipped 1 previous similar message [ 5433.791676] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5436.748050] Lustre: server umount fs26924-OST0000 complete [ 5439.907406] Lustre: server umount fs26924-OST0001 complete [ 5446.115782] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5446.115906] LustreError: 301904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5446.116273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5446.117296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5446.119162] Lustre: Skipped 1 previous similar message [ 5446.123303] LustreError: 301904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 5449.686057] Lustre: server umount lustre-OST0000 complete [ 5451.233123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5451.233958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5451.236420] Lustre: Skipped 1 previous similar message [ 5451.240438] Lustre: Skipped 2 previous similar messages [ 5456.352084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5456.355197] Lustre: Skipped 1 previous similar message [ 5457.052675] Lustre: server umount lustre-MDT0000 complete [ 5458.333165] LustreError: 300060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784409865 with bad export cookie 8540768162160511691 [ 5458.336468] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5458.337548] LustreError: 300060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5458.465853] Lustre: server umount lustre-MDT0001 complete [ 5462.005277] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 5463.044644] Key type lgssc unregistered [ 5463.173330] LNet: 313326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5463.175874] LNetError: 313326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5463.190385] LNet: Removed LNI 192.168.203.154@tcp [ 5463.522111] Key type .llcrypt unregistered [ 5463.523314] Key type ._llcrypt unregistered [ 5474.084186] Key type ._llcrypt registered [ 5474.086163] Key type .llcrypt registered [ 5474.131334] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 5480.399279] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5480.817734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5480.832310] alg: No test for adler32 (adler32-zlib) [ 5481.699456] Lustre: Lustre: Build Version: 2.17.54_174_g53e1600 [ 5481.803190] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 5483.391154] Key type lgssc registered [ 5483.794462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5495.486106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5495.490972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5496.585730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5496.596695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5496.636291] Lustre: lustre-MDT0000: new disk, initializing [ 5496.658704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5496.664442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5498.126168] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5502.362194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5502.412476] Lustre: 317156:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5502.441839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5502.443785] Lustre: Skipped 1 previous similar message [ 5502.481760] Lustre: lustre-MDT0001: new disk, initializing [ 5502.520678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5502.535522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5502.539289] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5503.981635] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5507.699585] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5509.539102] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5511.824794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5511.930121] Lustre: lustre-OST0000: new disk, initializing [ 5511.931828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5511.934407] Lustre: 319124:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5511.960503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5511.997295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5512.000899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5512.044924] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5513.988026] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5517.734584] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5518.419369] Lustre: Mounted lustre-client [ 5518.769806] Lustre: Unmounted lustre-client [ 5522.400123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5522.400338] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5522.402400] LustreError: Skipped 1 previous similar message [ 5522.408404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5522.411837] Lustre: Skipped 1 previous similar message [ 5523.426037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5523.428179] Lustre: Skipped 1 previous similar message [ 5527.891969] Lustre: server umount lustre-OST0000 complete [ 5533.151898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5533.154793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5533.158290] Lustre: Skipped 1 previous similar message [ 5533.160144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5535.390804] Lustre: server umount lustre-MDT0000 complete [ 5536.684608] LustreError: 317148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784409943 with bad export cookie 4348296979141034244 [ 5536.688230] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5536.688305] LustreError: 317148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5536.814017] Lustre: server umount lustre-MDT0001 complete [ 5540.463365] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 5541.437336] Key type lgssc unregistered [ 5574.367342] LNet: 321282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5574.369814] LNetError: 321282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5650.150622] LNet: Removed LNI 192.168.203.154@tcp [ 5650.469820] Key type .llcrypt unregistered [ 5650.471605] Key type ._llcrypt unregistered [ 5656.431071] Key type ._llcrypt registered [ 5656.432271] Key type .llcrypt registered [ 5656.469530] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 5662.878323] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5663.169397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5663.180335] alg: No test for adler32 (adler32-zlib) [ 5664.040847] Lustre: Lustre: Build Version: 2.17.54_174_g53e1600 [ 5664.136962] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 5665.719156] Key type lgssc registered [ 5666.092379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5676.068446] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 17:28:02 (1784410082) [ 5678.261689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5678.267520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5679.356388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5679.367613] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5679.404364] Lustre: lustre-MDT0000: new disk, initializing [ 5679.427066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5679.433392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5680.754740] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5684.904388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5684.947499] Lustre: 324480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5684.974462] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5684.976467] Lustre: Skipped 1 previous similar message [ 5685.011863] Lustre: lustre-MDT0001: new disk, initializing [ 5685.047614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5685.062217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5685.067830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5686.368204] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5690.057832] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5691.874385] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5694.145991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5694.252829] Lustre: lustre-OST0000: new disk, initializing [ 5694.254581] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5694.257119] Lustre: 326446:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5694.287414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5696.253832] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5696.320102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5696.323943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5696.367672] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5699.929941] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5706.720739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5706.721386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5706.723371] LustreError: Skipped 1 previous similar message [ 5706.728237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5706.730348] Lustre: Skipped 1 previous similar message [ 5711.633190] Lustre: server umount lustre-OST0000 complete [ 5715.936209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5715.940240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5715.946746] Lustre: Skipped 1 previous similar message [ 5715.949850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5716.960910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5716.961749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5719.160790] Lustre: server umount lustre-MDT0000 complete [ 5720.667383] LustreError: 324473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784410127 with bad export cookie 5580092134843392922 [ 5720.674771] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5720.831719] Lustre: server umount lustre-MDT0001 complete [ 5725.192925] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 5726.862551] Key type lgssc unregistered [ 5727.037565] LNet: 328356:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5727.041638] LNetError: 328356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5727.055509] LNet: Removed LNI 192.168.203.154@tcp [ 5727.560719] Key type .llcrypt unregistered [ 5727.563253] Key type ._llcrypt unregistered [ 5736.015973] Key type ._llcrypt registered [ 5736.017215] Key type .llcrypt registered [ 5736.061631] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5736.634810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5736.641405] alg: No test for adler32 (adler32-zlib) [ 5737.562187] Lustre: Lustre: Build Version: 2.17.54_174_g53e1600 [ 5737.685711] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 5739.287218] Key type lgssc registered [ 5739.658165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5743.116307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5743.128098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5744.313496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5746.041291] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5749.844280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5749.985654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5751.562612] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5754.989639] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5757.147256] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5759.918080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5760.074979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5762.421229] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5765.383918] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5779.225287] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5779.767050] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5782.076798] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.54@tcp (stopping) [ 5785.062881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5785.067492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5785.071924] Lustre: Skipped 2 previous similar messages [ 5785.077311] Lustre: Skipped 1 previous similar message [ 5787.181568] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.54@tcp (stopping) [ 5787.764838] Lustre: server umount lustre-MDT0000 complete [ 5789.398416] LustreError: 329587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784410196 with bad export cookie 9889481932918248999 [ 5789.405503] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5789.406707] LustreError: 329587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5790.175982] LustreError: 332097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5790.176194] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5790.180023] LustreError: 332097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5790.185187] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5792.301114] LustreError: 329600:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5794.207846] LustreError: 330895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5794.208105] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5794.214100] Lustre: Skipped 1 previous similar message [ 5795.557778] Lustre: server umount lustre-MDT0001 complete [ 5802.832885] Lustre: server umount lustre-OST0000 complete [ 5805.712911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5807.062667] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5808.045201] Lustre: 333806:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.154@tcp [ 5808.778239] Lustre: server umount MGS complete [ 5811.496592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5811.666634] LustreError: 334376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5811.688778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5812.953216] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5815.600750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5815.728221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5816.999160] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5819.681698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5821.731396] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5824.158396] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5824.761034] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 5826.817868] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5827.351507] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5831.142239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5831.152429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5831.152910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5831.162254] Lustre: Skipped 1 previous similar message [ 5834.764191] Lustre: server umount lustre-OST0000 complete [ 5836.256873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5836.257697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5836.267344] Lustre: Skipped 2 previous similar messages [ 5842.146425] Lustre: server umount lustre-MDT0000 complete [ 5843.465252] LustreError: 336770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784410250 with bad export cookie 9889481932918249825 [ 5843.468000] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5843.468501] LustreError: 336770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5843.574497] Lustre: server umount lustre-MDT0001 complete [ 5847.399151] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 5848.518797] Key type lgssc unregistered [ 5848.631383] LNet: 337808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5848.633437] LNetError: 337808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5848.643287] LNet: Removed LNI 192.168.203.154@tcp [ 5848.929700] Key type .llcrypt unregistered [ 5848.930943] Key type ._llcrypt unregistered [ 5855.424163] Key type ._llcrypt registered [ 5855.425256] Key type .llcrypt registered [ 5855.465229] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 5861.659791] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5862.031808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5862.083297] alg: No test for adler32 (adler32-zlib) [ 5862.940536] Lustre: Lustre: Build Version: 2.17.54_174_g53e1600 [ 5863.041535] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 5864.623175] Key type lgssc registered [ 5864.989253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5876.669841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5876.678432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5877.765150] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5877.775056] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5877.810501] Lustre: lustre-MDT0000: new disk, initializing [ 5877.834322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5877.840106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5879.446252] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5883.732401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5883.779193] Lustre: 340941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5883.805956] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5883.808550] Lustre: Skipped 1 previous similar message [ 5883.845194] Lustre: lustre-MDT0001: new disk, initializing [ 5883.881244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5883.894216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5883.898449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5885.230289] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5888.927110] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5890.699288] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5893.079761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5893.187944] Lustre: lustre-OST0000: new disk, initializing [ 5893.190959] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5893.194513] Lustre: 342908:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5893.222232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5895.158178] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5899.446401] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5900.279888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5900.282874] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5900.313050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5900.918302] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5900.979539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5902.347650] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5902.470314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5905.378216] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5905.378568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5905.379856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5905.382143] LustreError: Skipped 1 previous similar message [ 5909.547408] Lustre: server umount lustre-OST0000 complete [ 5914.592548] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5914.597414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5914.605226] Lustre: Skipped 1 previous similar message [ 5914.608843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5914.612705] Lustre: Skipped 1 previous similar message [ 5915.617364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5915.618468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5917.393040] Lustre: server umount lustre-MDT0000 complete [ 5919.164747] LustreError: 340934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784410325 with bad export cookie 16967332843210550413 [ 5919.169336] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5919.173164] LustreError: 340934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5919.331469] Lustre: server umount lustre-MDT0001 complete [ 5922.338637] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 17:32:08 (1784410328) [ 5928.463296] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 5931.545490] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5944.495793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5944.630746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5944.644397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5944.688266] Lustre: lustre-MDT0000: new disk, initializing [ 5944.717983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5944.724639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5946.519234] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5950.664764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5950.717231] Lustre: 347657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5950.747385] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5950.750266] Lustre: Skipped 1 previous similar message [ 5950.788755] Lustre: lustre-MDT0001: new disk, initializing [ 5950.821416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5950.833116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5950.837037] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5952.297305] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5956.369482] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5958.465611] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5961.244804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5961.369903] Lustre: lustre-OST0000: new disk, initializing [ 5961.371605] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5961.375461] Lustre: 349628:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5961.401514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5963.651669] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5967.859873] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5969.398949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5969.402591] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5969.432320] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5984.738620] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5984.738626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5984.739558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5984.746831] Lustre: Skipped 2 previous similar messages [ 5989.070710] Lustre: server umount lustre-OST0000 complete [ 5991.903905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5991.908140] LustreError: Skipped 1 previous similar message [ 5991.910623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5991.916627] Lustre: Skipped 1 previous similar message [ 5991.919670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5996.442655] Lustre: server umount lustre-MDT0000 complete [ 5997.743412] LustreError: 347649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784410404 with bad export cookie 16967332843210551295 [ 5997.746834] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5997.746871] LustreError: 347649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5997.869491] Lustre: server umount lustre-MDT0001 complete [ 6001.376637] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6002.371517] Key type lgssc unregistered [ 6002.494377] LNet: 351775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6002.496834] LNetError: 351775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6002.504317] LNet: Removed LNI 192.168.203.154@tcp [ 6002.805897] Key type .llcrypt unregistered [ 6002.807277] Key type ._llcrypt unregistered [ 6035.291358] Key type ._llcrypt registered [ 6035.292459] Key type .llcrypt registered [ 6035.333354] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6035.429444] Key type .llcrypt unregistered [ 6035.430688] Key type ._llcrypt unregistered [ 6063.541898] Key type ._llcrypt registered [ 6063.543021] Key type .llcrypt registered [ 6063.579757] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6063.680186] Key type .llcrypt unregistered [ 6063.681310] Key type ._llcrypt unregistered [ 6074.821684] Key type ._llcrypt registered [ 6074.822647] Key type .llcrypt registered [ 6074.861073] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6074.963357] Key type .llcrypt unregistered [ 6074.964430] Key type ._llcrypt unregistered [ 6086.185341] Key type ._llcrypt registered [ 6086.186453] Key type .llcrypt registered [ 6086.224224] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6086.327302] Key type .llcrypt unregistered [ 6086.328346] Key type ._llcrypt unregistered [ 6113.633336] Key type ._llcrypt registered [ 6113.634345] Key type .llcrypt registered [ 6113.670357] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6113.763232] Key type .llcrypt unregistered [ 6113.764314] Key type ._llcrypt unregistered [ 6125.814985] Key type ._llcrypt registered [ 6125.816095] Key type .llcrypt registered [ 6125.853287] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6125.958166] Key type .llcrypt unregistered [ 6125.959251] Key type ._llcrypt unregistered [ 6137.797311] Key type ._llcrypt registered [ 6137.798592] Key type .llcrypt registered [ 6137.847399] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6137.971263] Key type .llcrypt unregistered [ 6137.973188] Key type ._llcrypt unregistered [ 6149.897451] Key type ._llcrypt registered [ 6149.898657] Key type .llcrypt registered [ 6149.952580] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6150.074257] Key type .llcrypt unregistered [ 6150.076235] Key type ._llcrypt unregistered [ 6163.017967] Key type ._llcrypt registered [ 6163.020106] Key type .llcrypt registered [ 6163.064600] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6163.171225] Key type .llcrypt unregistered [ 6163.172922] Key type ._llcrypt unregistered [ 6176.179258] Key type ._llcrypt registered [ 6176.181212] Key type .llcrypt registered [ 6176.222631] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6176.330226] Key type .llcrypt unregistered [ 6176.331447] Key type ._llcrypt unregistered [ 6197.721174] Key type ._llcrypt registered [ 6197.722562] Key type .llcrypt registered [ 6197.762206] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6197.859267] Key type .llcrypt unregistered [ 6197.860422] Key type ._llcrypt unregistered [ 6208.360134] Key type ._llcrypt registered [ 6208.361242] Key type .llcrypt registered [ 6208.403426] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6208.497244] Key type .llcrypt unregistered [ 6208.498258] Key type ._llcrypt unregistered [ 6220.578303] Key type ._llcrypt registered [ 6220.580243] Key type .llcrypt registered [ 6220.618230] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6220.721161] Key type .llcrypt unregistered [ 6220.722247] Key type ._llcrypt unregistered [ 6264.629578] Key type ._llcrypt registered [ 6264.632845] Key type .llcrypt registered [ 6264.687247] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 6272.751587] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 6273.371735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6273.396271] alg: No test for adler32 (adler32-zlib) [ 6274.306399] Lustre: Lustre: Build Version: 2.17.54_174_g53e1600 [ 6274.442894] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 6276.047161] Key type lgssc registered [ 6276.563645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6292.080457] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6292.091637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6293.195891] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 6293.209347] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 6293.248762] Lustre: tst250fs-MDT0000: new disk, initializing [ 6293.275221] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6293.284386] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6294.773205] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6299.167039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6299.224143] Lustre: 371684:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6299.252903] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 6299.255388] Lustre: Skipped 1 previous similar message [ 6299.297612] Lustre: tst250fs-MDT0001: new disk, initializing [ 6299.337839] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6299.351747] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6299.355949] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6300.864858] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6303.374365] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6305.877817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6306.036336] Lustre: tst250fs-OST0000: new disk, initializing [ 6306.039271] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 6306.043667] Lustre: 373316:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6306.092267] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6308.327755] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6312.450113] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6312.453580] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6312.497231] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6312.960110] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6313.034915] Lustre: tst250fs-OST0001: new disk, initializing [ 6313.036630] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 6313.039073] Lustre: 374189:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6313.082277] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6315.320152] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6318.047821] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 6321.010525] Lustre: 371687:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.154@tcp: IR log tst250fs-mdtir failed, not fatal: rc = -2 [ 6328.801353] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6328.804148] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 6328.808437] Lustre: Skipped 3 previous similar messages [ 6328.810879] Lustre: Skipped 2 previous similar messages [ 6333.922896] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 6333.925927] Lustre: Skipped 4 previous similar messages [ 6334.614588] Lustre: server umount tst250fs-MDT0000 complete [ 6335.982441] LustreError: 373315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784410742 with bad export cookie 9587698642536574539 [ 6335.986831] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6335.987876] LustreError: 373315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6336.119759] Lustre: server umount tst250fs-MDT0001 complete [ 6347.288382] Lustre: server umount tst250fs-OST0000 complete [ 6359.069934] Lustre: server umount tst250fs-OST0001 complete [ 6366.286183] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 6369.306834] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 6380.497328] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 17:39:46 (1784410786) [ 6381.055646] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 6386.876790] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 6390.864082] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 6402.660787] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6156 sec ========== 17:40:08 (1784410808) [ 6403.236976] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:40:09 (1784410809) === [ 6404.414673] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:40:10 (1784410810) === [ 6418.864225] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6420.035835] Key type lgssc unregistered [ 6420.184389] LNet: 383288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6420.188533] LNetError: 383288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6420.197779] LNet: Removed LNI 192.168.203.154@tcp [ 6420.581154] Key type .llcrypt unregistered [ 6420.583212] Key type ._llcrypt unregistered