[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 575771824 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002404] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004020] kvm-guest: setup PV IPIs [ 0.007295] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010032] pid_max: default: 32768 minimum: 301 [ 0.011142] LSM: Security Framework initializing [ 0.012079] Yama: becoming mindful. [ 0.013000] SELinux: Initializing. [ 0.013100] *** VALIDATE selinux *** [ 0.022323] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027026] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029029] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030131] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032144] *** VALIDATE tmpfs *** [ 0.034107] *** VALIDATE proc *** [ 0.035282] *** VALIDATE cgroup *** [ 0.036014] *** VALIDATE cgroup2 *** [ 0.037309] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039074] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041040] Spectre V2 : User space: Vulnerable [ 0.042011] Speculative Store Bypass: Vulnerable [ 0.045473] debug: unmapping init [mem 0xffffffff9d459000-0xffffffff9d460fff] [ 0.048140] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049784] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050034] ... version: 2 [ 0.051016] ... bit width: 48 [ 0.052017] ... generic registers: 4 [ 0.053017] ... value mask: 0000ffffffffffff [ 0.054021] ... max period: 00007fffffffffff [ 0.055022] ... fixed-purpose events: 3 [ 0.056015] ... event mask: 000000070000000f [ 0.057341] rcu: Hierarchical SRCU implementation. [ 0.059749] smp: Bringing up secondary CPUs ... [ 0.060617] x86: Booting SMP configuration: [ 0.061050] .... node #0, CPUs: #1 #2 #3 [ 0.074088] smp: Brought up 1 node, 4 CPUs [ 0.076016] smpboot: Max logical packages: 1 [ 0.077015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.104407] node 0 deferred pages initialised in 25ms [ 0.108126] devtmpfs: initialized [ 0.110330] x86/mm: Memory block size: 128MB [ 0.116627] gcov: version magic: 0x41383552 [ 0.121339] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.122085] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.124327] pinctrl core: initialized pinctrl subsystem [ 0.127308] [ 0.127930] ************************************************************* [ 0.130018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.133018] ** ** [ 0.136016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.138020] ** ** [ 0.141016] ** This means that this kernel is built to expose internal ** [ 0.144018] ** IOMMU data structures, which may compromise security on ** [ 0.148019] ** your system. ** [ 0.152022] ** ** [ 0.155058] ** If you see this message and you are not debugging the ** [ 0.157015] ** kernel, report this immediately to your vendor! ** [ 0.160020] ** ** [ 0.164020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166022] ************************************************************* [ 0.171292] NET: Registered protocol family 16 [ 0.174223] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.177117] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.182083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.186117] cpuidle: using governor menu [ 0.188578] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.191730] PCI: Using configuration type 1 for base access [ 0.194140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.206142] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.207053] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.210053] cryptd: max_cpu_qlen set to 1000 [ 0.214309] ACPI: Added _OSI(Module Device) [ 0.216015] ACPI: Added _OSI(Processor Device) [ 0.217027] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.219022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.224931] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.234640] ACPI: Interpreter enabled [ 0.235074] ACPI: PM: (supports S0 S3 S4 S5) [ 0.237016] ACPI: Using IOAPIC for interrupt routing [ 0.239147] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.245418] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.257534] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.259054] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.262026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.267093] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.272520] acpiphp: Slot [2] registered [ 0.274249] acpiphp: Slot [5] registered [ 0.275182] acpiphp: Slot [6] registered [ 0.278216] acpiphp: Slot [7] registered [ 0.280141] acpiphp: Slot [8] registered [ 0.281144] acpiphp: Slot [9] registered [ 0.282173] acpiphp: Slot [10] registered [ 0.283000] acpiphp: Slot [3] registered [ 0.283208] acpiphp: Slot [4] registered [ 0.284000] acpiphp: Slot [11] registered [ 0.286120] acpiphp: Slot [12] registered [ 0.288187] acpiphp: Slot [13] registered [ 0.290582] acpiphp: Slot [14] registered [ 0.292149] acpiphp: Slot [15] registered [ 0.294119] acpiphp: Slot [16] registered [ 0.296245] acpiphp: Slot [17] registered [ 0.298561] acpiphp: Slot [18] registered [ 0.300204] acpiphp: Slot [19] registered [ 0.302800] acpiphp: Slot [20] registered [ 0.305138] acpiphp: Slot [21] registered [ 0.307118] acpiphp: Slot [22] registered [ 0.309138] acpiphp: Slot [23] registered [ 0.311153] acpiphp: Slot [24] registered [ 0.313128] acpiphp: Slot [25] registered [ 0.315099] acpiphp: Slot [26] registered [ 0.317136] acpiphp: Slot [27] registered [ 0.319128] acpiphp: Slot [28] registered [ 0.321139] acpiphp: Slot [29] registered [ 0.324114] acpiphp: Slot [30] registered [ 0.325981] acpiphp: Slot [31] registered [ 0.327091] PCI host bridge to bus 0000:00 [ 0.328031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.331038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.334031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.338030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.342039] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.344039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.348190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.351189] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.355454] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.367810] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.374070] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.377023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.381024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.384025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.387644] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.388807] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.392053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.396012] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.403019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.416028] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.423021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.429460] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.440018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.448018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.480021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.495431] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.504027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.512024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.532019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.541000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.554020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.564024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.596019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.615190] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.627022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.636020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.653019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.668339] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.680019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.686018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.704017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.719336] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.728020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.740019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.776018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.802492] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.806435] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.809462] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.812743] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.815312] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.821197] iommu: Default domain type: Passthrough [ 0.824315] SCSI subsystem initialized [ 0.826259] ACPI: bus type USB registered [ 0.828283] usbcore: registered new interface driver usbfs [ 0.832111] usbcore: registered new interface driver hub [ 0.834110] usbcore: registered new device driver usb [ 0.837297] pps_core: LinuxPPS API ver. 1 registered [ 0.839013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.843101] PTP clock support registered [ 0.845142] EDAC MC: Ver: 3.0.0 [ 0.847562] PCI: Using ACPI for IRQ routing [ 0.851788] NetLabel: Initializing [ 0.854015] NetLabel: domain hash size = 128 [ 0.856012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.858304] NetLabel: unlabeled traffic allowed by default [ 0.863139] vgaarb: loaded [ 0.865371] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.868021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.876162] clocksource: Switched to clocksource kvm-clock [ 1.034329] VFS: Disk quotas dquot_6.6.0 [ 1.038489] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.044282] *** VALIDATE ramfs *** [ 1.045775] *** VALIDATE hugetlbfs *** [ 1.048612] pnp: PnP ACPI init [ 1.051586] pnp: PnP ACPI: found 6 devices [ 1.071743] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.078321] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.082455] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.087587] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.090983] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.095255] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.100681] NET: Registered protocol family 2 [ 1.103857] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.118146] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.125324] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.135670] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.142354] TCP: Hash tables configured (established 65536 bind 65536) [ 1.147506] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.152459] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.156892] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.162420] NET: Registered protocol family 1 [ 1.166188] RPC: Registered named UNIX socket transport module. [ 1.170712] RPC: Registered udp transport module. [ 1.173281] RPC: Registered tcp transport module. [ 1.176301] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.179870] NET: Registered protocol family 44 [ 1.182810] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.185651] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.189723] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.192310] PCI: CLS 0 bytes, default 64 [ 1.194436] Unpacking initramfs... [ 3.057338] debug: unmapping init [mem 0xffff92117cc54000-0xffff92117ffbffff] [ 3.064920] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.067580] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.072656] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.605614] Initialise system trusted keyrings [ 3.609295] Key type blacklist registered [ 3.611758] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.621042] zbud: loaded [ 3.624896] *** VALIDATE nfs *** [ 3.626888] *** VALIDATE nfs4 *** [ 3.630791] pstore: using deflate compression [ 3.635830] Platform Keyring initialized [ 3.780656] NET: Registered protocol family 38 [ 3.783219] Key type asymmetric registered [ 3.785377] Asymmetric key parser 'x509' registered [ 3.787815] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.792793] io scheduler mq-deadline registered [ 3.796582] io scheduler kyber registered [ 3.801195] io scheduler bfq registered [ 3.805425] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.812986] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.819397] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.823949] ACPI: Power Button [PWRF] [ 3.837593] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.850181] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.874509] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.887370] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.907816] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.939446] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.977060] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.986423] Non-volatile memory driver v1.3 [ 3.990304] Linux agpgart interface v0.103 [ 4.042258] virtio_blk virtio1: [vda] 150040 512-byte logical blocks (76.8 MB/73.3 MiB) [ 4.047153] vda: detected capacity change from 0 to 76820480 [ 4.074659] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.078710] vdb: detected capacity change from 0 to 1073741824 [ 4.100173] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.104853] vdc: detected capacity change from 0 to 2621440000 [ 4.130883] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.138163] vdd: detected capacity change from 0 to 2621440000 [ 4.170277] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.175880] vde: detected capacity change from 0 to 4294967296 [ 4.205277] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.208490] vdf: detected capacity change from 0 to 4294967296 [ 4.217197] libphy: Fixed MDIO Bus: probed [ 4.230427] usbcore: registered new interface driver usbserial_generic [ 4.234499] usbserial: USB Serial support registered for generic [ 4.238608] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.244800] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.248941] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.252156] mousedev: PS/2 mouse device common for all mice [ 4.259144] rtc_cmos 00:05: RTC can wake from S4 [ 4.266658] rtc_cmos 00:05: registered as rtc0 [ 4.268420] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.273529] intel_pstate: CPU model not supported [ 4.275210] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.281888] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.283588] hid: raw HID events driver (C) Jiri Kosina [ 4.287515] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.294538] usbcore: registered new interface driver usbhid [ 4.296958] usbhid: USB HID core driver [ 4.298789] drop_monitor: Initializing network drop monitor service [ 4.301641] Initializing XFRM netlink socket [ 4.303779] NET: Registered protocol family 10 [ 4.307187] Segment Routing with IPv6 [ 4.308757] NET: Registered protocol family 17 [ 4.311103] mpls_gso: MPLS GSO support [ 4.317262] RAS: Correctable Errors collector initialized. [ 4.319528] AVX version of gcm_enc/dec engaged. [ 4.321294] AES CTR mode by8 optimization enabled [ 4.419695] sched_clock: Marking stable (4419603315, 0)->(5538436554, -1118833239) [ 4.423779] registered taskstats version 1 [ 4.426249] Loading compiled-in X.509 certificates [ 4.428505] zswap: loaded using pool lzo/zbud [ 4.461440] Key type big_key registered [ 4.481964] Key type encrypted registered [ 4.483808] ima: No TPM chip found, activating TPM-bypass! [ 4.486623] ima: Allocated hash algorithm: sha1 [ 4.488454] ima: No architecture policies found [ 4.490757] evm: Initialising EVM extended attributes: [ 4.492586] evm: security.selinux [ 4.494107] evm: security.ima [ 4.495302] evm: security.capability [ 4.496765] evm: HMAC attrs: 0x1 [ 4.499420] rtc_cmos 00:05: setting system clock to 2026-09-07 01:52:09 UTC (1788745929) [ 4.505984] debug: unmapping init [mem 0xffffffff9e403000-0xffffffff9e5fffff] [ 4.509757] debug: unmapping init [mem 0xffffffff9d182000-0xffffffff9d458fff] [ 4.519132] Write protecting the kernel read-only data: 28672k [ 4.523738] debug: unmapping init [mem 0xffffffff9b803000-0xffffffff9b9fffff] [ 4.527038] debug: unmapping init [mem 0xffffffff9c114000-0xffffffff9c1fffff] [ 4.571250] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.582226] systemd[1]: Detected virtualization kvm. [ 4.584707] systemd[1]: Detected architecture x86-64. [ 4.586812] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.629730] systemd[1]: No hostname configured. [ 4.631776] systemd[1]: Set hostname to . [ 4.637080] random: systemd: uninitialized urandom read (16 bytes read) [ 4.639827] systemd[1]: Initializing machine ID from random generator. [ 4.775132] random: ln: uninitialized urandom read (6 bytes read) [ 4.888317] random: systemd: uninitialized urandom read (16 bytes read) [ 4.892237] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.901450] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.909486] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.140260] device-mapper: uevent: version 1.0.3 [ 6.144134] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.615804] random: fast init done [ 7.782582] scsi host0: ata_piix [ 7.791527] virtio_net virtio0 ens2: renamed from eth0 [ 7.855274] scsi host1: ata_piix [ 7.865308] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.884048] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.010820] random: crng init done [ 13.013273] random: 7 urandom warning(s) missed due to ratelimiting [ 13.906340] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 14.931565] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.251793] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.663631] SELinux: Disabled at runtime. [ 17.743102] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.754844] systemd[1]: Detected virtualization kvm. [ 17.760613] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.906359] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.914800] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.925514] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.934941] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.939311] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.955263] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.966570] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ 19.205892] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 20.400508] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.309264] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.443831] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.919981] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.175777] EDAC sbridge: Ver: 1.1.2 [ 26.034155] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 26.745606] NFS: Registering the id_resolver key type [ 26.747910] Key type id_resolver registered [ 26.757194] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg641-server login: [ 88.342123] libcfs: loading out-of-tree module taints kernel. [ 88.364857] Key type ._llcrypt registered [ 88.366492] Key type .llcrypt registered [ 88.482800] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 107.754025] hrtimer: interrupt took 2061534 ns [ 110.327927] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 112.628546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 112.645758] alg: No test for adler32 (adler32-zlib) [ 114.082381] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 114.894258] LNet: Added LNI 192.168.206.141@tcp [8/256/0/180] [ 116.687236] Key type lgssc registered [ 119.146405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 142.823383] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 189.175215] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 204.769928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 204.862512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 206.226853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 206.290533] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 206.521304] Lustre: lustre-MDT0000: new disk, initializing [ 206.716644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 206.770479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 212.089590] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.367078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 226.545370] Lustre: 6519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 226.605538] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 226.618302] Lustre: Skipped 1 previous similar message [ 226.784507] Lustre: lustre-MDT0001: new disk, initializing [ 226.882270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 226.913462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 226.926805] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 232.009343] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.056953] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 248.346248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 248.648743] Lustre: lustre-OST0000: new disk, initializing [ 248.652598] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 248.659621] Lustre: 8456:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 248.739589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 250.381363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 250.388036] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 250.494862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 256.343852] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 270.934673] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 271.086597] Lustre: lustre-OST0001: new disk, initializing [ 271.089758] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 271.095523] Lustre: 9528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 271.154221] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 277.004618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 277.018830] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 277.112416] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 277.809862] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 291.157739] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 298.544801] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 305.370806] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing check_logdir /tmp/testlogs/ [ 311.151823] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing yml_node [ 316.884473] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 319.484256] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 322.024521] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 324.283967] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 6 21:57:26 EDT 2026 [ 342.950868] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 345.259228] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 364.003807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.007927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 364.028607] Lustre: Skipped 1 previous similar message [ 364.045525] Lustre: Skipped 3 previous similar messages [ 369.122547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 369.137361] Lustre: Skipped 3 previous similar messages [ 374.247460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 374.257668] Lustre: Skipped 3 previous similar messages [ 376.287425] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 378.578244] Lustre: server umount lustre-MDT0000 complete [ 383.816559] LustreError: 10344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746308 with bad export cookie 13561965867386711510 [ 383.821753] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 383.835927] LustreError: 10344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 384.485025] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 384.488399] LustreError: 9816:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 384.490364] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 384.506855] Lustre: Skipped 3 previous similar messages [ 384.549481] LustreError: 9816:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 6 previous similar messages [ 388.511714] LustreError: 6526:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 388.513314] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 388.521082] LustreError: 6526:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 388.526573] Lustre: Skipped 2 previous similar messages [ 393.632138] LustreError: 6526:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 393.652431] LustreError: 6526:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 398.303155] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 398.600492] Lustre: server umount lustre-MDT0001 complete [ 403.159247] Lustre: server umount lustre-OST0000 complete [ 407.498992] Lustre: server umount lustre-OST0001 complete [ 415.342631] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 425.020427] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 465.938878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 466.196468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 466.231439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 466.316084] Lustre: lustre-MDT0000: new disk, initializing [ 466.435461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 466.480580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 470.335553] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 481.747372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.978275] Lustre: 15087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 482.026696] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 482.029846] Lustre: Skipped 1 previous similar message [ 482.095358] Lustre: lustre-MDT0001: new disk, initializing [ 482.236280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 482.284382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 482.297878] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 487.003955] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 497.323848] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 503.513689] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 510.452708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 510.875732] Lustre: lustre-OST0000: new disk, initializing [ 510.883754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 510.893982] Lustre: 17058:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 511.023901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 512.240132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 512.254586] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 512.389511] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 517.758964] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 528.177860] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 533.546865] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 533.799327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 538.770367] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 539.048671] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 543.238202] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 543.243817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 543.250807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 543.252598] Lustre: Skipped 1 previous similar message [ 543.260255] Lustre: Skipped 2 previous similar messages [ 547.600992] Lustre: server umount lustre-OST0000 complete [ 553.444112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 557.912426] Lustre: server umount lustre-MDT0000 complete [ 558.560229] LustreError: 16966:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.585949] LustreError: 16966:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 561.617301] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746486 with bad export cookie 13561965867386713050 [ 561.620775] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 561.633095] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 562.059501] Lustre: server umount lustre-MDT0001 complete [ 567.544068] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 22:01:30 (1788746490) [ 584.982413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.489346] LustreError: 19945:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.587617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 589.951505] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 595.937118] LustreError: 19946:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.957640] LustreError: 19946:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 598.783784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 604.051945] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 610.759576] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 616.177207] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 618.287015] Lustre: Failing over lustre-MDT0000 [ 618.493465] Lustre: server umount lustre-MDT0000 complete [ 619.491404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 619.504918] LustreError: Skipped 1 previous similar message [ 619.508310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 619.524272] Lustre: Skipped 1 previous similar message [ 619.535056] LustreError: 19946:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 637.407279] Lustre: 3648:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788746546/real 1788746546] req@ffff9210c3401880 x1875636166757376/t0(0) o400->MGC192.168.206.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788746562 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 637.453291] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 638.047284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 647.648414] LustreError: 3646:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff921201509180 x1875636166760192/t0(0) o250->MGC192.168.206.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 647.976692] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 652.483102] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 653.285758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 653.289421] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 653.318504] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 661.144721] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 667.619452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 667.627551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 667.637977] Lustre: Skipped 2 previous similar messages [ 667.653676] Lustre: Skipped 3 previous similar messages [ 679.392490] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 679.521376] Lustre: server umount lustre-MDT0000 complete [ 679.908766] LustreError: 19941:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 679.921834] LustreError: 19941:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 17 previous similar messages [ 682.788058] LustreError: 20680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746607 with bad export cookie 13561965867386714464 [ 682.800896] LustreError: 20680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 682.803268] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 683.128790] Lustre: server umount lustre-MDT0001 complete [ 692.031934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 692.539186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 692.544191] Lustre: Skipped 2 previous similar messages [ 696.864854] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 704.326361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 709.451657] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 717.094591] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 722.499399] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 724.485387] Lustre: Failing over lustre-MDT0000 [ 724.647499] Lustre: server umount lustre-MDT0000 complete [ 725.476032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 725.482750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 742.353836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 742.503718] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 742.763712] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 746.516798] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 748.008057] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 748.017715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 748.031497] Lustre: Skipped 1 previous similar message [ 748.050921] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 754.875389] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 758.384885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 758.397985] Lustre: Skipped 1 previous similar message [ 758.402737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 758.404807] Lustre: Skipped 4 previous similar messages [ 763.366760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 764.844915] Lustre: server umount lustre-MDT0000 complete [ 767.981279] LustreError: 24857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746692 with bad export cookie 13561965867386715402 [ 767.984725] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 767.990902] LustreError: 24857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 768.021127] LustreError: 3649:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting [ 768.227419] Lustre: server umount lustre-MDT0001 complete [ 775.011187] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 22:04:58 (1788746698) [ 789.614958] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 799.000302] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 830.928652] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 838.555575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 838.733728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 838.760498] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 838.866783] Lustre: lustre-MDT0000: new disk, initializing [ 838.928819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 838.940600] Lustre: Skipped 2 previous similar messages [ 838.961804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 842.767035] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 852.180556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 852.285985] Lustre: 29655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 852.563624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 856.652036] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 865.243895] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 869.970330] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 876.083437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 876.309810] Lustre: 31625:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 877.649101] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 877.700195] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 881.777820] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 883.205583] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 883.229306] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 883.235736] Lustre: Skipped 1 previous similar message [ 883.242541] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 889.488928] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 893.821764] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 913.894904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 913.897737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 913.911716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 913.927691] Lustre: Skipped 2 previous similar messages [ 913.936829] Lustre: Skipped 3 previous similar messages [ 915.185899] Lustre: server umount lustre-OST0000 complete [ 924.366975] Lustre: server umount lustre-MDT0000 complete [ 927.212494] LustreError: 29647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746852 with bad export cookie 13561965867386715808 [ 927.227255] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 927.532457] Lustre: server umount lustre-MDT0001 complete [ 935.511210] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 937.805667] Key type lgssc unregistered [ 938.072397] LNet: 33600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.081229] LNetError: 33600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 938.093455] LNet: Removed LNI 192.168.206.141@tcp [ 938.778150] Key type .llcrypt unregistered [ 938.779992] Key type ._llcrypt unregistered [ 958.755972] Key type ._llcrypt registered [ 958.758264] Key type .llcrypt registered [ 958.827938] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 959.623198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 959.676067] alg: No test for adler32 (adler32-zlib) [ 960.704258] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 960.899913] LNet: Added LNI 192.168.206.141@tcp [8/256/0/180] [ 962.536585] Key type lgssc registered [ 963.693773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 969.814335] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 969.831390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 971.142750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 974.547696] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 981.257707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 981.525668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 984.738689] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 990.495576] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 994.857867] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 999.819564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1000.101280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1004.728910] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 1006.252507] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1010.429710] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1030.770338] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 22:09:14 (1788746954) [ 1032.295793] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1033.915835] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1038.480465] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 22:09:21 (1788746961) [ 1044.384679] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 22:09:27 (1788746967) [ 1045.179480] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1051.174179] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1051.177302] Lustre: Skipped 2 previous similar messages [ 1052.353354] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 22:09:35 (1788746975) [ 1057.856178] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1057.862116] Lustre: Skipped 4 previous similar messages [ 1067.449510] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1067.458749] Lustre: Skipped 5 previous similar messages [ 1069.553413] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 22:09:52 (1788746992) [ 1071.903344] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1074.508685] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1081.419723] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 22:10:04 (1788747004) [ 1083.560700] Lustre: Disabling parameter general.jobid_name= in log params [ 1083.564390] Lustre: Skipped 4 previous similar messages [ 1091.882723] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 22:10:15 (1788747015) [ 1103.841044] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1103.845811] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.855909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1104.867310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1104.870637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1104.880473] Lustre: Skipped 1 previous similar message [ 1106.447588] Lustre: server umount lustre-OST0000 complete [ 1109.475631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1109.483273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1109.491892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1115.104729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1115.108141] Lustre: Skipped 2 previous similar messages [ 1120.224133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1120.226841] Lustre: Skipped 1 previous similar message [ 1123.295120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1123.430372] Lustre: server umount lustre-MDT0000 complete [ 1125.345775] LustreError: 35007:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1125.361333] LustreError: 35007:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 1125.672570] LustreError: 34989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747050 with bad export cookie 13182048463658749241 [ 1125.673906] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1125.686734] LustreError: 34989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1125.919391] Lustre: server umount lustre-MDT0001 complete [ 1132.868656] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 1134.734468] Key type lgssc unregistered [ 1134.969332] LNet: 42867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1134.976426] LNetError: 42867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1134.986574] LNet: Removed LNI 192.168.206.141@tcp [ 1135.567065] Key type .llcrypt unregistered [ 1135.569663] Key type ._llcrypt unregistered [ 1146.886365] Key type ._llcrypt registered [ 1146.887777] Key type .llcrypt registered [ 1146.963967] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 1158.113321] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 1158.686634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1158.713274] alg: No test for adler32 (adler32-zlib) [ 1159.672961] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 1159.909630] LNet: Added LNI 192.168.206.141@tcp [8/256/0/180] [ 1161.575744] Key type lgssc registered [ 1162.503725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1187.589688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1187.619290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1188.844224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1188.879672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1188.969521] Lustre: lustre-MDT0000: new disk, initializing [ 1189.086037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1189.102271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1192.755105] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 1200.684867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1200.771838] Lustre: 45991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1200.814767] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1200.818478] Lustre: Skipped 1 previous similar message [ 1200.874221] Lustre: lustre-MDT0001: new disk, initializing [ 1200.955905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1201.000089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1201.009721] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1204.029632] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 1210.899128] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1214.730496] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1219.678171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1219.861631] Lustre: lustre-OST0000: new disk, initializing [ 1219.864405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1219.868925] Lustre: 47957:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1219.943266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1224.447265] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 1226.797296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1226.807933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1226.871646] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1230.845605] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1234.199298] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1234.423589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1237.846693] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1238.005356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1242.089091] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1242.089385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1242.101929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1242.118143] Lustre: Skipped 1 previous similar message [ 1245.649427] Lustre: server umount lustre-OST0000 complete [ 1252.322100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1252.324839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1252.326181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1252.326187] Lustre: Skipped 1 previous similar message [ 1252.344992] Lustre: Skipped 1 previous similar message [ 1254.388888] Lustre: server umount lustre-MDT0000 complete [ 1256.739436] LustreError: 45984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747181 with bad export cookie 9873285445155368793 [ 1256.744349] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1256.745013] LustreError: 45984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1256.927302] Lustre: server umount lustre-MDT0001 complete [ 1262.713567] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:13:06 (1788747186) [ 1267.337473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1267.709632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1270.588090] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 1272.800876] LustreError: 50415:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1272.812359] LustreError: 50415:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1275.709331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1275.959791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1278.648767] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 1283.399341] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1287.118513] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1291.405373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1291.635104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1295.823855] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 1300.692278] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1304.299965] LustreError: 52941:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1304.854442] Lustre: Modifying parameter general.timeout=1 in log params [ 1306.066128] Lustre: Modifying parameter general.timeout=3 in log params [ 1306.069958] Lustre: Skipped 1 previous similar message [ 1308.268129] Lustre: Modifying parameter general.timeout=7 in log params [ 1308.271060] Lustre: Skipped 3 previous similar messages [ 1312.732025] Lustre: Modifying parameter general.timeout=15 in log params [ 1312.735171] Lustre: Skipped 7 previous similar messages [ 1321.011072] Lustre: Modifying parameter general.timeout=28 in log params [ 1321.013826] Lustre: Skipped 12 previous similar messages [ 1337.578713] Lustre: Modifying parameter general.timeout=53 in log params [ 1337.581020] Lustre: Skipped 24 previous similar messages [ 1369.772051] Lustre: Modifying parameter general.timeout=102 in log params [ 1369.778463] Lustre: Skipped 48 previous similar messages [ 1395.173186] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 22:15:18 (1788747318) [ 1397.129545] Lustre: 59872:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1398.376365] Lustre: 59970:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1400.838890] Lustre: 60168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1400.847895] Lustre: 60168:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1422.444547] Lustre: 61714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1422.452570] Lustre: 61714:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1422.464834] LustreError: 61714:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1425.663274] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 22:15:48 (1788747348) [ 1434.593123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1434.599879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1434.614829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1434.621166] Lustre: Skipped 1 previous similar message [ 1438.690609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1438.690643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1438.696096] Lustre: Skipped 1 previous similar message [ 1438.709122] Lustre: Skipped 1 previous similar message [ 1442.237571] Lustre: server umount lustre-MDT0000 complete [ 1443.811518] LustreError: 50411:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1443.823522] LustreError: 50411:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1444.399551] LustreError: 52257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747369 with bad export cookie 9873285445155369675 [ 1444.404621] LustreError: 52257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1444.406785] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1444.576343] Lustre: server umount lustre-MDT0001 complete [ 1456.272223] Lustre: server umount lustre-OST0000 complete [ 1477.064230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1477.348322] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1477.374700] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1477.382901] Lustre: Skipped 50 previous similar messages [ 1477.560570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1481.316895] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 1487.835585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1487.922825] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1487.946409] Lustre: 64243:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1492.187366] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 1498.913947] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1502.711572] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1508.117764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1508.315429] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1513.510567] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 1519.532768] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1521.252172] Lustre: 66807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1531.360566] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1531.372882] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1531.397472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1531.407676] Lustre: Skipped 1 previous similar message [ 1534.762832] Lustre: server umount lustre-OST0000 complete [ 1539.553489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1539.560872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1539.569779] Lustre: Skipped 1 previous similar message [ 1539.573390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1539.577709] Lustre: Skipped 1 previous similar message [ 1544.947955] Lustre: server umount lustre-MDT0000 complete [ 1547.234039] LustreError: 65303:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1547.256295] LustreError: 65303:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1547.793561] LustreError: 64233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747472 with bad export cookie 9873285445155372349 [ 1547.798245] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1547.813285] LustreError: 64233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1548.093218] Lustre: server umount lustre-MDT0001 complete [ 1556.451071] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 1558.476187] Key type lgssc unregistered [ 1558.808768] LNet: 68015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1558.816465] LNetError: 68015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1558.838703] LNet: Removed LNI 192.168.206.141@tcp [ 1559.575434] Key type .llcrypt unregistered [ 1559.577445] Key type ._llcrypt unregistered [ 1578.686804] Key type ._llcrypt registered [ 1578.688966] Key type .llcrypt registered [ 1578.787363] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 1579.405773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1579.507889] alg: No test for adler32 (adler32-zlib) [ 1580.523846] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 1580.735670] LNet: Added LNI 192.168.206.141@tcp [8/256/0/180] [ 1582.410916] Key type lgssc registered [ 1583.244439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1589.440737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1589.464173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1590.814683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1594.759956] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 1601.001250] LustreError: 69402:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1601.016196] LustreError: 69402:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1601.726331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1602.155329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1606.027888] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 1612.546667] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1616.859475] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1622.041942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1622.493848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1626.861308] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 1631.789146] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1641.294415] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1648.097713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1648.100583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1648.105937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1652.193725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1652.195979] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1653.811733] Lustre: server umount lustre-MDT0000 complete [ 1657.243671] LustreError: 69382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747582 with bad export cookie 2879499385456293198 [ 1657.263621] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1657.270170] LustreError: 69382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1657.325876] LustreError: 69397:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1657.329229] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1657.356178] Lustre: Skipped 1 previous similar message [ 1657.360887] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1657.364329] Lustre: Skipped 2 previous similar messages [ 1657.533404] Lustre: server umount lustre-MDT0001 complete [ 1666.871316] Lustre: server umount lustre-OST0000 complete [ 1684.527953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1684.714367] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1684.737677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1684.901219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1687.517083] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 1693.030984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1693.094528] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1693.118552] Lustre: 74280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1693.169183] LustreError: 74291:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1693.439739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1696.106078] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 1700.908890] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1704.497259] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1709.206716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1709.337906] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1709.406263] LustreError: 76128:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1709.580388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1713.324505] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 1718.399466] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1719.859387] Lustre: 76855:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1721.430992] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1730.033227] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1730.037726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1730.040486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1730.048577] LustreError: Skipped 1 previous similar message [ 1730.076130] Lustre: Skipped 1 previous similar message [ 1732.049814] Lustre: server umount lustre-OST0000 complete [ 1739.233589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1739.243845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1739.257435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1739.267897] Lustre: Skipped 1 previous similar message [ 1742.290693] Lustre: server umount lustre-MDT0000 complete [ 1745.227488] LustreError: 74271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747670 with bad export cookie 2879499385456294031 [ 1745.233643] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1745.242139] LustreError: 74271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1745.482189] Lustre: server umount lustre-MDT0001 complete [ 1752.562984] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 1754.461406] Key type lgssc unregistered [ 1754.724694] LNet: 78063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1754.738429] LNetError: 78063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1754.756022] LNet: Removed LNI 192.168.206.141@tcp [ 1755.467942] Key type .llcrypt unregistered [ 1755.471802] Key type ._llcrypt unregistered [ 1773.576533] Key type ._llcrypt registered [ 1773.578298] Key type .llcrypt registered [ 1773.656611] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 1774.285711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1774.343592] alg: No test for adler32 (adler32-zlib) [ 1775.346736] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 1775.516833] LNet: Added LNI 192.168.206.141@tcp [8/256/0/180] [ 1777.167225] Key type lgssc registered [ 1778.261767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1783.927849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1783.942980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1785.271763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1788.490553] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 1795.215980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1795.754509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1799.847306] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 1806.211394] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1811.297680] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1817.013348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1817.320935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1822.592712] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 1828.622669] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1838.632920] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1839.455734] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1840.879533] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1840.886207] Lustre: Skipped 1 previous similar message [ 1843.049828] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1843.052581] Lustre: Skipped 2 previous similar messages [ 1847.589251] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1847.592986] Lustre: Skipped 6 previous similar messages [ 1856.138451] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1856.142973] Lustre: Skipped 14 previous similar messages [ 1872.683427] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1872.687133] Lustre: Skipped 23 previous similar messages [ 1904.721247] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1904.723844] Lustre: Skipped 39 previous similar messages [ 1920.899834] Lustre: 87541:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1923.142931] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 22:24:06 (1788747846) [ 1924.693886] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1926.493926] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 22:24:09 (1788747849) [ 1933.284215] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1933.290959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1933.307626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1934.307094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1934.307560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1934.316799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1934.344317] Lustre: Skipped 1 previous similar message [ 1936.449709] Lustre: server umount lustre-OST0000 complete [ 1944.032150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1944.036173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1944.052410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1949.664403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1949.669091] Lustre: Skipped 2 previous similar messages [ 1954.271201] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1954.629832] Lustre: server umount lustre-MDT0000 complete [ 1954.788959] LustreError: 79450:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1954.814363] LustreError: 79450:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 1958.767642] LustreError: 79430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747883 with bad export cookie 8027754932506400402 [ 1958.771947] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1958.783803] LustreError: 79430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1959.121626] Lustre: server umount lustre-MDT0001 complete [ 1969.716847] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 1972.343689] Key type lgssc unregistered [ 1972.608987] LNet: 88791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1972.619825] LNetError: 88791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1972.644749] LNet: Removed LNI 192.168.206.141@tcp [ 1973.419046] Key type .llcrypt unregistered [ 1973.420950] Key type ._llcrypt unregistered [ 1977.314223] Key type ._llcrypt registered [ 1977.318787] Key type .llcrypt registered [ 1977.510919] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1982.400370] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules [ 1983.527138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1983.612721] alg: No test for adler32 (adler32-zlib) [ 1984.607314] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 1984.898473] LNet: Added LNI 192.168.206.141@tcp [8/256/0/180] [ 1986.623222] Key type lgssc registered [ 1987.696988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1988.713418] LustreError: 89991:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2005.212885] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 2010.343611] LustreError: 89991:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2013.910275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2013.935316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2015.269541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2018.936507] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 2025.446060] LustreError: 91016:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2025.461858] LustreError: 91016:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2026.511230] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 22:25:49 (1788747949) [ 2029.665921] LustreError: 91048:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2029.677076] LustreError: 91048:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2029.686278] LustreError: 91048:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 2029.932301] Lustre: server umount lustre-MDT0000 complete [ 2041.518348] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 2043.985638] Key type lgssc unregistered [ 2044.332739] LNet: 92541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2044.354114] LNetError: 92541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2044.385895] LNet: Removed LNI 192.168.206.141@tcp [ 2045.165699] Key type .llcrypt unregistered [ 2045.169843] Key type ._llcrypt unregistered [ 2057.622231] Key type ._llcrypt registered [ 2057.624258] Key type .llcrypt registered [ 2057.696590] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 2058.342914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2058.350773] alg: No test for adler32 (adler32-zlib) [ 2059.382293] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 2059.550765] LNet: Added LNI 192.168.206.141@tcp [8/256/0/180] [ 2061.175349] Key type lgssc registered [ 2062.241653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2069.873899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2069.886322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2071.449747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2076.980825] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 2081.762469] LustreError: 93735:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2081.777843] LustreError: 93735:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2084.335812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2084.884654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2089.003284] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 2095.053473] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2099.309365] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2104.771890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2105.308437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2110.718951] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 2116.385593] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2157.570859] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 22:28:00 (1788748080) [ 2159.071848] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2160.352779] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 22:28:03 (1788748083) [ 2166.755410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2166.759780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2166.777184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2167.780993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2167.787038] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2170.042206] Lustre: server umount lustre-MDT0000 complete [ 2172.899805] LustreError: 93729:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.916042] LustreError: 93729:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2173.081891] LustreError: 93714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748098 with bad export cookie 13000524473177481272 [ 2173.087345] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2173.093819] LustreError: 93714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2173.380317] Lustre: server umount lustre-MDT0001 complete [ 2187.355853] Lustre: server umount lustre-OST0000 complete [ 2197.444339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2197.778854] LustreError: 98003:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2197.915204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2201.319470] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 2203.108567] LustreError: 98004:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2207.205509] LustreError: 98003:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2207.855609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2208.309265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2212.170553] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 2219.791612] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2225.482940] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2237.827521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2237.978307] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2237.983576] LustreError: 97993:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2237.997409] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2238.002873] LustreError: 100088:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2238.007366] LustreError: 100088:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2238.010154] LustreError: 100088:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2238.061744] Lustre: server umount lustre-OST0000 complete [ 2238.064572] LustreError: 100088:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2244.231118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2244.403905] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2244.409212] LustreError: 97991:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2244.414828] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2244.423952] LustreError: 100549:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2244.432982] LustreError: 100549:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2244.438295] LustreError: 100549:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2244.511510] Lustre: server umount lustre-OST0000 complete [ 2244.513872] LustreError: 100549:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2253.014183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2253.133219] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2253.135185] Lustre: Found index 0 for lustre-OST0000, updating log [ 2253.138730] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2253.178306] Lustre: lustre-OST0000: new disk, initializing [ 2253.180260] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2253.183612] Lustre: 101083:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2253.322372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2258.380578] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 2261.447152] Lustre: Failing over lustre-OST0000 [ 2261.758578] Lustre: server umount lustre-OST0000 complete [ 2265.450671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2265.454590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2265.459952] Lustre: Skipped 2 previous similar messages [ 2265.465653] Lustre: Skipped 2 previous similar messages [ 2270.688889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2270.700847] Lustre: Skipped 2 previous similar messages [ 2271.795717] Lustre: server umount lustre-MDT0000 complete [ 2275.798223] LustreError: 97985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748200 with bad export cookie 13000524473177484016 [ 2275.811745] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2275.812404] LustreError: 97985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2285.224047] Lustre: server umount lustre-MDT0001 complete [ 2292.935868] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 22:30:16 (1788748216) [ 2309.313713] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 2318.803738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2319.039387] LustreError: 103432:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2319.101527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2322.752535] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 2331.567658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2336.242359] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 2339.257682] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2345.534805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2346.143136] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2351.273388] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 2356.402741] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2356.438942] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2356.439659] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2358.095321] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2358.184382] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2358.297511] Lustre: lustre-OST0001: new disk, initializing [ 2358.305878] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2358.316055] Lustre: 105750:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2358.635303] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2358.644917] Lustre: Skipped 2 previous similar messages [ 2361.997557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2362.006757] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2362.123028] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2364.637532] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 2372.369509] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2383.581265] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2387.868556] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.41@tcp (stopping) [ 2388.962920] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2388.970907] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2393.724845] Lustre: server umount lustre-MDT0001 complete [ 2395.115912] LustreError: 104162:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.149873] LustreError: 104162:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 2401.613371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2401.712155] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2401.715635] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2401.743403] Lustre: 103420:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2403.088977] LustreError: 107306:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2403.105237] LustreError: 107306:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2403.116198] LustreError: 103424:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2403.203952] LustreError: 107321:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.141@tcp: cfg command failed: rc = -17 [ 2403.216271] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2403.216271] [ 2405.973992] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 2407.413113] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2413.610780] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 22:32:16 (1788748336) [ 2431.036226] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 2450.034101] Lustre: 109362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2472.552817] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2475.907509] Lustre: 110496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2550.752212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2550.752629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2550.752992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2550.752997] Lustre: Skipped 5 previous similar messages [ 2550.763585] Lustre: Skipped 3 previous similar messages [ 2556.457830] Lustre: server umount lustre-MDT0000 complete [ 2559.577023] LustreError: 103414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748484 with bad export cookie 13000524473177484947 [ 2559.581156] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2559.588174] LustreError: 103414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2559.893464] Lustre: server umount lustre-MDT0001 complete [ 2573.607302] Lustre: server umount lustre-OST0000 complete [ 2576.872963] Lustre: 93044:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788748485/real 1788748485] req@ffff9210ccfff100 x1875638206414592/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788748501 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2576.907547] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2576.921176] Lustre: Skipped 2 previous similar messages [ 2577.261560] Lustre: server umount lustre-OST0001 complete [ 2581.946239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2589.895532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2597.787439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2606.009971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2623.796625] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 2633.044667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2633.098885] Lustre: lustre-MDT0000: reset Object Index mappings [ 2633.501226] LustreError: 114444:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2633.508081] LustreError: 114444:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 2633.554578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2633.558359] Lustre: Skipped 1 previous similar message [ 2637.552680] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 2646.356282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2646.385217] Lustre: lustre-MDT0001: reset Object Index mappings [ 2651.584776] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 2654.410423] Lustre: 115590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2660.347906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2666.780328] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 2671.279950] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:231 to 0x280000400:257) [ 2673.984199] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2678.585788] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2678.594744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:350 to 0x280000401:386) [ 2678.642082] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2679.664484] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 2686.469854] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2689.741340] Lustre: 117466:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2733.087739] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 22:37:36 (1788748656) [ 2760.677174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2760.684996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2760.694409] Lustre: Skipped 2 previous similar messages [ 2760.712868] Lustre: Skipped 10 previous similar messages [ 2765.972140] Lustre: server umount lustre-MDT0000 complete [ 2769.041162] LustreError: 115592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748694 with bad export cookie 13000524473177531812 [ 2769.062081] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2770.918338] LustreError: 114440:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2770.935479] LustreError: 114440:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 2775.669642] Lustre: server umount lustre-MDT0001 complete [ 2785.688268] Lustre: server umount lustre-OST0000 complete [ 2795.436931] Lustre: server umount lustre-OST0001 complete [ 2802.509585] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 2811.223171] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 2844.417466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2844.592387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2844.610942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2844.695781] Lustre: lustre-MDT0000: new disk, initializing [ 2844.803750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2844.811938] Lustre: Skipped 3 previous similar messages [ 2844.827084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2848.525318] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 2854.832298] Lustre: server umount lustre-MDT0000 complete [ 2863.580135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2863.888049] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2863.910642] Lustre: 121738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2863.922235] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2868.685039] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 2872.663530] Lustre: server umount lustre-MDT0000 complete [ 2879.873627] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 22:40:03 (1788748803) [ 2881.340709] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2882.610690] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 22:40:06 (1788748806) [ 2897.239918] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 2905.250307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2908.703471] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 2914.842262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2914.935359] Lustre: 123789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2914.993827] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2914.997423] Lustre: Skipped 1 previous similar message [ 2915.046307] Lustre: lustre-MDT0001: new disk, initializing [ 2915.149470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2915.162282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2918.238353] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 2921.874203] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2926.791092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2926.993961] Lustre: lustre-OST0000: new disk, initializing [ 2926.996932] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2927.005175] Lustre: 125363:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2928.861555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2928.870549] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2928.935868] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2931.582608] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 2939.375273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2939.490426] Lustre: lustre-OST0001: new disk, initializing [ 2939.492816] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2939.497078] Lustre: 126235:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2940.988304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2940.996516] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2941.122830] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2943.827284] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 2951.178663] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2954.138892] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3208.093406] Lustre: 132628:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3208.103841] Lustre: 132628:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3208.109921] Lustre: 132628:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3208.113765] Lustre: 132628:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3208.117691] Lustre: 132628:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3208.121579] Lustre: 132628:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3389.201566] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:48:32 (1788749312) [ 3402.210461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3402.212403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3402.215392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3402.215402] Lustre: Skipped 8 previous similar messages [ 3402.216149] Lustre: Skipped 4 previous similar messages [ 3407.526665] Lustre: server umount lustre-MDT0000 complete [ 3409.253053] LustreError: 123782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749334 with bad export cookie 13000524473177598221 [ 3409.257663] LustreError: 123782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3409.259314] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3412.448561] LustreError: 127051:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3412.457705] LustreError: 127051:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 3415.578195] Lustre: server umount lustre-MDT0001 complete [ 3423.824892] Lustre: server umount lustre-OST0000 complete [ 3440.095229] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3440.199800] Lustre: server umount lustre-OST0001 complete [ 3443.546366] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 3448.380241] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 3464.435877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3464.566995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3464.582495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3464.632718] Lustre: lustre-MDT0000: new disk, initializing [ 3464.673449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3464.677910] Lustre: Skipped 5 previous similar messages [ 3464.690079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3466.950522] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 3472.257528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3472.308881] Lustre: 135659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3472.431171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3474.302070] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 3478.908372] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3481.505193] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3484.473667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3484.606259] Lustre: 137628:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3486.226118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3486.232432] Lustre: Skipped 1 previous similar message [ 3486.235850] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3486.274963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3487.358454] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 3492.252709] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3496.366804] Lustre: lustre-MDD0000: changelog on [ 3497.528897] Lustre: lustre-MDD0001: changelog on [ 4482.701366] Lustre: lustre-MDD0001: changelog off [ 4484.475826] Lustre: 135668:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4485.355155] Lustre: lustre-MDD0000: changelog off [ 4494.304971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4494.315373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4494.338947] Lustre: Skipped 4 previous similar messages [ 4494.348079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4494.352513] Lustre: Skipped 11 previous similar messages [ 4494.816524] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4497.709738] Lustre: server umount lustre-OST0000 complete [ 4501.483841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4505.059916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4505.069424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4505.075957] Lustre: Skipped 2 previous similar messages [ 4505.094048] Lustre: Skipped 3 previous similar messages [ 4507.451472] Lustre: server umount lustre-MDT0000 complete [ 4510.179506] LustreError: 136656:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4510.199601] LustreError: 136656:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 4510.946574] LustreError: 137627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750435 with bad export cookie 13000524473178408632 [ 4510.947799] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4510.953973] LustreError: 137627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4511.263385] Lustre: server umount lustre-MDT0001 complete [ 4519.741490] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 4523.103509] Key type lgssc unregistered [ 4523.466857] LNet: 146624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4523.479618] LNetError: 146624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4523.505849] LNet: Removed LNI 192.168.206.141@tcp [ 4524.821311] Key type .llcrypt unregistered [ 4524.823595] Key type ._llcrypt unregistered [ 4545.319198] Key type ._llcrypt registered [ 4545.320482] Key type .llcrypt registered [ 4545.421610] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 4560.635976] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 4561.774470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4561.809858] alg: No test for adler32 (adler32-zlib) [ 4562.804629] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 4563.068396] LNet: Added LNI 192.168.206.141@tcp [8/256/0/180] [ 4564.823192] Key type lgssc registered [ 4566.098765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4599.735698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4599.776200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4601.088223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4601.142245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4601.221707] Lustre: lustre-MDT0000: new disk, initializing [ 4601.335867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4601.363597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4606.497485] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 4616.697526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4616.805140] Lustre: 149823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4616.878880] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4616.884712] Lustre: Skipped 1 previous similar message [ 4616.973464] Lustre: lustre-MDT0001: new disk, initializing [ 4617.064146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4617.090333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4617.102371] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4621.178979] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 4630.510106] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4635.323321] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4640.689433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4640.889353] Lustre: lustre-OST0000: new disk, initializing [ 4640.893445] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4640.899661] Lustre: 151789:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4640.984377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4646.898281] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 4650.571896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4650.598910] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4650.700877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4655.847298] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4660.497845] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_module obdecho/obdecho [ 4662.115135] LustreError: 152792:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4662.123458] LustreError: 152792:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 4663.816056] LustreError: 152888:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 4673.506572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4673.515129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4673.523611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4676.580465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4676.581914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4676.590776] Lustre: Skipped 1 previous similar message [ 4676.607648] Lustre: Skipped 1 previous similar message [ 4679.446702] Lustre: server umount lustre-MDT0000 complete [ 4682.288417] LustreError: 149815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750607 with bad export cookie 6844918142023657154 [ 4682.295226] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4682.298166] LustreError: 149815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4682.497373] Lustre: server umount lustre-MDT0001 complete [ 4695.910436] Lustre: server umount lustre-OST0000 complete [ 4718.811754] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 4730.820818] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 4770.170650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4770.434929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4770.485679] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4770.606620] Lustre: lustre-MDT0000: new disk, initializing [ 4770.707979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4770.730433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4774.560131] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 4784.281521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4784.352545] Lustre: 156608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4784.421551] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4784.428329] Lustre: Skipped 1 previous similar message [ 4784.485066] Lustre: lustre-MDT0001: new disk, initializing [ 4784.551060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4784.589787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4784.596376] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4789.941506] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 4799.843365] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4805.338784] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4812.005384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4812.416763] Lustre: lustre-OST0000: new disk, initializing [ 4812.424188] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4812.429213] Lustre: 158576:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4812.598728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4818.554558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4818.568125] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4818.650589] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4819.002853] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 4827.372253] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4831.763522] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4832.036803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4837.004504] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4837.209444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4839.394273] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4839.398794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4839.435915] Lustre: Skipped 1 previous similar message [ 4839.445166] Lustre: Skipped 1 previous similar message [ 4844.513247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4844.526032] Lustre: Skipped 2 previous similar messages [ 4845.422530] Lustre: server umount lustre-OST0000 complete [ 4851.169131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4851.186788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4851.216521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4856.188174] Lustre: server umount lustre-MDT0000 complete [ 4859.521752] LustreError: 157401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750784 with bad export cookie 6844918142023658113 [ 4859.527036] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4859.532034] LustreError: 157401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4859.760818] Lustre: server umount lustre-MDT0001 complete [ 4862.603784] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 23:13:05 (1788750785) [ 4863.956779] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4865.645426] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 23:13:08 (1788750788) [ 4872.476102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4872.883850] LustreError: 160987:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4872.914020] LustreError: 160987:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 4873.000117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4877.236518] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 4878.307719] LustreError: 160988:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4882.401301] LustreError: 160987:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4885.476913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4890.110210] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 4897.331605] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4903.884220] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4911.761830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4912.338735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4912.352958] Lustre: Skipped 1 previous similar message [ 4919.300527] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 4925.755263] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4936.161339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4936.174017] Lustre: Skipped 3 previous similar messages [ 4936.176372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4936.176379] Lustre: Skipped 2 previous similar messages [ 4937.402704] Lustre: server umount lustre-MDT0000 complete [ 4940.194776] LustreError: 160968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750865 with bad export cookie 6844918142023658995 [ 4940.203962] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4940.211895] LustreError: 160968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4941.286944] LustreError: 160984:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4945.312411] LustreError: 160984:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4945.332828] LustreError: 160984:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 4946.702723] Lustre: server umount lustre-MDT0001 complete [ 4952.341784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4964.062745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4964.357371] LustreError: 164828:0:(llog.c:1655:llog_backup()) MGC192.168.206.141@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4964.365886] Lustre: 164828:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.141@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4967.354755] LustreError: 164833:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4985.318062] LustreError: 164833:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4985.339175] LustreError: 164833:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 13 previous similar messages [ 5018.492048] LustreError: 164833:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5018.511242] LustreError: 164833:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 24 previous similar messages [ 5025.759758] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5025.765356] Lustre: 164828:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.141@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5066.304073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5067.307945] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5071.060530] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5079.296881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5082.595939] LustreError: 162815:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5082.608529] LustreError: 162815:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 44 previous similar messages [ 5091.985176] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5093.360585] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5097.945245] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5099.314555] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5103.901329] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5105.254187] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5122.263247] Lustre: server umount lustre-MDT0000 complete [ 5124.066742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5124.069124] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5124.090181] Lustre: Skipped 3 previous similar messages [ 5125.014920] LustreError: 160968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751050 with bad export cookie 6844918142023665449 [ 5125.023168] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5125.028042] LustreError: 160968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5125.312483] Lustre: server umount lustre-MDT0001 complete [ 5138.596357] Lustre: server umount lustre-OST0000 complete [ 5155.503258] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 5163.205821] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 5195.028843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5195.288634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5195.352156] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5195.518190] Lustre: lustre-MDT0000: new disk, initializing [ 5195.631204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5195.636791] Lustre: Skipped 1 previous similar message [ 5195.675983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5200.083524] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5209.578770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5209.713626] Lustre: 169929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5210.007478] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5214.448311] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5222.176651] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5226.244427] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5231.435139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5231.645337] Lustre: lustre-OST0000: new disk, initializing [ 5231.651508] Lustre: Skipped 1 previous similar message [ 5231.658205] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5231.662755] Lustre: Skipped 2 previous similar messages [ 5231.669868] Lustre: 171897:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5236.971974] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5239.899386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5239.907513] Lustre: Skipped 1 previous similar message [ 5239.918129] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5239.990709] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5244.213193] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5248.129255] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5248.394626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5251.652945] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5251.812802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5255.661525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5255.665425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5255.675592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5255.679739] Lustre: Skipped 6 previous similar messages [ 5259.205293] Lustre: server umount lustre-OST0000 complete [ 5276.127611] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5276.250984] Lustre: server umount lustre-MDT0000 complete [ 5278.942905] LustreError: 169921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751203 with bad export cookie 6844918142023682277 [ 5278.947639] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5278.955466] LustreError: 169921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5279.180976] Lustre: server umount lustre-MDT0001 complete [ 5284.954833] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 23:20:08 (1788751208) [ 5289.562437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5289.886168] LustreError: 174310:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5289.901877] LustreError: 174310:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 7 previous similar messages [ 5293.317331] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5299.313106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5302.657404] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5308.077828] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5312.316045] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5318.649308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5324.401432] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5330.961245] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5343.660187] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 23:21:07 (1788751267) [ 5349.866233] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5349.870870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5349.873326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5349.873333] Lustre: Skipped 6 previous similar messages [ 5349.877738] LustreError: Skipped 2 previous similar messages [ 5349.887783] Lustre: Skipped 4 previous similar messages [ 5352.665316] Lustre: server umount lustre-OST0000 complete [ 5361.838154] Lustre: server umount lustre-MDT0000 complete [ 5364.688875] LustreError: 174292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751289 with bad export cookie 6844918142023683159 [ 5364.692389] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5364.696862] LustreError: 174292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5365.007526] Lustre: server umount lustre-MDT0001 complete [ 5374.844318] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 5377.180639] Key type lgssc unregistered [ 5377.404098] LNet: 178033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5377.410642] LNetError: 178033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5377.440127] LNet: Removed LNI 192.168.206.141@tcp [ 5378.231125] Key type .llcrypt unregistered [ 5378.232597] Key type ._llcrypt unregistered [ 5392.142411] Key type ._llcrypt registered [ 5392.144384] Key type .llcrypt registered [ 5392.267802] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 5392.972214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5393.033490] alg: No test for adler32 (adler32-zlib) [ 5394.105287] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 5394.297623] LNet: Added LNI 192.168.206.141@tcp [8/256/0/180] [ 5395.935530] Key type lgssc registered [ 5396.861646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5403.395563] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5403.420426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5418.976727] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5424.095551] LustreError: 179252:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.141@tcp: failed processing log, type 4: rc = -110 [ 5429.215639] LustreError: 179252:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5429.222617] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5429.225396] LustreError: MGC192.168.206.141@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5429.232749] LustreError: 179252:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5429.239483] LustreError: 179252:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 5429.244959] LustreError: 179252:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 5429.289716] Lustre: server umount lustre-OST0000 complete [ 5429.294531] LustreError: 179252:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5434.414234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5435.698164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5438.972868] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5445.093237] LustreError: 179752:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5445.107327] LustreError: 179752:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5445.369730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5445.643561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5448.923825] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5454.992646] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5459.118298] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5464.160213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5464.333877] LustreError: 181567:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5464.340881] Lustre: 181567:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.141@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5464.503108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5468.834644] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5473.989748] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5475.817207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5475.822434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5475.823747] Lustre: Skipped 2 previous similar messages [ 5475.834306] Lustre: Skipped 2 previous similar messages [ 5480.929817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5480.937504] Lustre: Skipped 2 previous similar messages [ 5481.574406] Lustre: server umount lustre-MDT0000 complete [ 5483.861178] LustreError: 180486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751408 with bad export cookie 7957026040430771062 [ 5483.862327] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5483.868016] LustreError: 180486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5484.078667] Lustre: server umount lustre-MDT0001 complete [ 5496.846888] Lustre: server umount lustre-OST0000 complete [ 5502.961746] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 5509.589263] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 5533.633081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5533.772565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5533.797221] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5533.863736] Lustre: lustre-MDT0000: new disk, initializing [ 5533.918748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5533.934862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5537.261847] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5544.962551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5545.053304] Lustre: 185087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5545.108193] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5545.113433] Lustre: Skipped 1 previous similar message [ 5545.184519] Lustre: lustre-MDT0001: new disk, initializing [ 5545.261664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5545.302530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5545.313834] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5548.548825] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5555.765578] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5559.686251] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5564.349387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5564.587608] Lustre: lustre-OST0000: new disk, initializing [ 5564.591943] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5564.599942] Lustre: 187053:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5564.674774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5569.668611] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5574.700313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5574.710573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5574.774594] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5577.114635] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5581.376257] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5581.562353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5584.641901] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5584.814453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5589.984305] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5589.989031] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5589.992353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5589.998938] Lustre: Skipped 1 previous similar message [ 5590.008393] Lustre: Skipped 1 previous similar message [ 5592.204201] Lustre: server umount lustre-OST0000 complete [ 5595.105260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5595.119402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5595.132127] Lustre: Skipped 1 previous similar message [ 5600.225459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5600.233535] Lustre: Skipped 1 previous similar message [ 5601.000362] Lustre: server umount lustre-MDT0000 complete [ 5604.513370] LustreError: 185080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751529 with bad export cookie 7957026040430771783 [ 5604.517727] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5604.519182] LustreError: 185080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5604.760939] Lustre: server umount lustre-MDT0001 complete [ 5611.374549] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 23:25:34 (1788751534) [ 5625.163667] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 5627.511731] Key type lgssc unregistered [ 5627.773129] LNet: 189995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5627.783776] LNetError: 189995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5627.794565] LNet: Removed LNI 192.168.206.141@tcp [ 5628.467865] Key type .llcrypt unregistered [ 5628.470393] Key type ._llcrypt unregistered [ 5642.318452] Key type ._llcrypt registered [ 5642.320346] Key type .llcrypt registered [ 5642.395474] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 5643.061391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5643.105344] alg: No test for adler32 (adler32-zlib) [ 5644.109587] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 5644.301783] LNet: Added LNI 192.168.206.141@tcp [8/256/0/180] [ 5645.959781] Key type lgssc registered [ 5646.920749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5653.237956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5653.261916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5654.619418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5658.216232] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5664.738517] LustreError: 191189:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5664.759635] LustreError: 191189:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5664.806303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5665.169235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5668.904710] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5674.154608] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5677.627355] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5683.190886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5683.360276] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5683.363465] LustreError: 193052:0:(llog.c:1696:llog_backup()) MGC192.168.206.141@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5683.369403] Lustre: 193052:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.141@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5683.455743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5688.773991] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5694.573786] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5701.088872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5701.096920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5701.103337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5701.107239] Lustre: Skipped 2 previous similar messages [ 5701.113238] Lustre: Skipped 1 previous similar message [ 5702.850918] Lustre: server umount lustre-MDT0000 complete [ 5705.868238] LustreError: 191170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751630 with bad export cookie 17570287766188790315 [ 5705.875669] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5705.882153] LustreError: 191170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5706.172664] Lustre: server umount lustre-MDT0001 complete [ 5719.696229] Lustre: server umount lustre-OST0000 complete [ 5725.973818] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 5733.868773] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 5762.069313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5762.222380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5762.242917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5762.296035] Lustre: lustre-MDT0000: new disk, initializing [ 5762.354174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5762.364262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5765.453962] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5773.355444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5773.451368] Lustre: 196575:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5773.483972] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5773.493359] Lustre: Skipped 1 previous similar message [ 5773.497234] LustreError: 197356:0:(llog.c:1696:llog_backup()) MGC192.168.206.141@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5773.507626] LustreError: 197356:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5773.515648] Lustre: 197356:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.141@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5773.544318] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5773.551161] Lustre: Skipped 1 previous similar message [ 5773.613439] Lustre: lustre-MDT0001: new disk, initializing [ 5773.686393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5773.715485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5773.720249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5777.052794] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5784.449624] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5788.496544] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5793.273026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5793.455307] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5793.459736] Lustre: Skipped 1 previous similar message [ 5793.466287] LustreError: 198518:0:(llog.c:1696:llog_backup()) MGC192.168.206.141@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5793.474094] LustreError: 198518:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5793.479072] Lustre: 198518:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.141@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5793.486622] Lustre: 198518:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5793.509943] Lustre: lustre-OST0000: new disk, initializing [ 5793.514122] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5793.525374] Lustre: 198542:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5793.595592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5798.022988] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5798.933334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5798.940934] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5799.031908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5805.107670] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5809.150065] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5809.318303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5812.947906] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5813.149350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5819.367510] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5819.371277] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5819.379774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5819.379781] Lustre: Skipped 1 previous similar message [ 5819.411329] Lustre: Skipped 1 previous similar message [ 5820.910832] Lustre: server umount lustre-OST0000 complete [ 5824.480408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5824.493112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5824.497258] Lustre: Skipped 1 previous similar message [ 5829.607022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5829.614835] Lustre: Skipped 1 previous similar message [ 5834.720414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5834.725988] Lustre: Skipped 1 previous similar message [ 5838.303123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5838.441846] Lustre: server umount lustre-MDT0000 complete [ 5839.842157] LustreError: 196585:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5839.854760] LustreError: 196585:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5841.234165] LustreError: 196566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751766 with bad export cookie 17570287766188791029 [ 5841.238404] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5841.244670] LustreError: 196566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5841.503327] Lustre: server umount lustre-MDT0001 complete [ 5848.624288] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 23:29:31 (1788751771) [ 5862.056810] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 5869.394283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5869.752351] LustreError: 201668:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5869.796508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5874.064227] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5875.177115] LustreError: 201669:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5879.269020] LustreError: 201668:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5880.822447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5884.677931] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5887.165911] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5893.343683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5899.144547] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5906.480987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5906.631768] Lustre: lustre-OST0001: new disk, initializing [ 5906.635738] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5906.647233] Lustre: 203974:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5906.736909] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5906.738932] Lustre: Skipped 2 previous similar messages [ 5912.056340] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5913.675686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5913.685692] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5913.770429] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5922.977281] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5931.285683] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5933.005662] Lustre: DEBUG MARKER: ADD OST3 [ 5937.098900] loop: module loaded [ 5937.146747] loop0: detected capacity change from 0 to 204800000 [ 5937.415472] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5937.432470] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5937.448938] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5942.365312] loop0: detected capacity change from 0 to 204800000 [ 5943.702904] Lustre: DEBUG MARKER: STOP OST3 [ 5946.770392] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5946.960368] Lustre: lustre-OST0003: new disk, initializing [ 5946.963965] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5946.969083] Lustre: 205911:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5948.378580] Lustre: Failing over lustre-OST0003 [ 5948.444412] Lustre: server umount lustre-OST0003 complete [ 5948.669162] LustreError: 203155:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5952.264442] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5960.746911] Lustre: DEBUG MARKER: START OST3 again [ 5966.269800] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5966.505960] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 5967.629881] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5967.658764] Lustre: lustre-OST0003: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5971.326906] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 5971.967260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5971.976518] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5971.995930] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 5991.048448] Lustre: Failing over lustre-OST0003 [ 5991.126643] Lustre: server umount lustre-OST0003 complete [ 5992.416816] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5992.421138] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5992.422461] LustreError: Skipped 1 previous similar message [ 5992.423085] LustreError: 203156:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5992.423092] LustreError: 203156:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 11 previous similar messages [ 5992.468316] Lustre: Skipped 2 previous similar messages [ 6009.315653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6009.325503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6009.359935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6014.597959] Lustre: server umount lustre-MDT0000 complete [ 6017.924597] LustreError: 201650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751942 with bad export cookie 17570287766188791911 [ 6017.936806] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6017.937892] LustreError: 201650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6018.023155] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6018.039318] Lustre: Skipped 3 previous similar messages [ 6027.179781] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6027.182952] Lustre: Skipped 8 previous similar messages [ 6027.189040] LustreError: 201663:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6027.216181] LustreError: 201663:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 13 previous similar messages [ 6032.351741] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6032.609757] Lustre: server umount lustre-MDT0001 complete [ 6036.389369] Lustre: server umount lustre-OST0000 complete [ 6039.626511] Lustre: server umount lustre-OST0001 complete [ 6045.289649] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 6053.069467] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 6084.207743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6084.449914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6084.469592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6084.537150] Lustre: lustre-MDT0000: new disk, initializing [ 6084.654180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6084.660095] Lustre: Skipped 2 previous similar messages [ 6084.716544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6088.625879] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 6097.376137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6097.485836] Lustre: 211431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6097.744372] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6101.251412] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 6109.395838] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6114.392515] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6119.805262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6120.039935] Lustre: lustre-OST0000: new disk, initializing [ 6120.042679] Lustre: Skipped 1 previous similar message [ 6120.050237] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6120.059665] Lustre: Skipped 2 previous similar messages [ 6120.066283] Lustre: 213399:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6121.989562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6122.000223] Lustre: Skipped 1 previous similar message [ 6122.005173] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6122.101383] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6125.493862] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 6133.794814] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6137.617816] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6137.836579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6142.023395] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6142.230858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6147.555445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6147.560420] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6147.572377] Lustre: Skipped 1 previous similar message [ 6147.573276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6147.573281] Lustre: Skipped 3 previous similar messages [ 6158.303126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6158.451954] Lustre: server umount lustre-OST0000 complete [ 6167.904739] Lustre: server umount lustre-MDT0000 complete [ 6168.034430] LustreError: 212334:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6168.046387] LustreError: 212334:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 6170.709052] LustreError: 211421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752095 with bad export cookie 17570287766188794193 [ 6170.711444] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6170.720213] LustreError: 211421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6171.040424] Lustre: server umount lustre-MDT0001 complete [ 6173.646943] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 23:34:57 (1788752097) [ 6185.733103] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 6193.519104] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 6220.485429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6220.720119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6220.754927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6220.853285] Lustre: lustre-MDT0000: new disk, initializing [ 6220.955816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6220.961735] Lustre: Skipped 2 previous similar messages [ 6220.983867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6224.298459] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 6232.232312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6232.340882] Lustre: 218006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6232.565763] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6236.018136] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 6242.872725] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6246.755110] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6251.264822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6251.441143] Lustre: 219975:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6253.111065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6255.386722] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 6261.429307] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6264.317722] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6264.477431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6267.648619] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6267.791573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6273.512474] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6273.512644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6273.514684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6273.514691] Lustre: Skipped 7 previous similar messages [ 6273.518280] LustreError: Skipped 1 previous similar message [ 6273.547219] Lustre: Skipped 4 previous similar messages [ 6275.189110] Lustre: server umount lustre-OST0000 complete [ 6283.890337] Lustre: server umount lustre-MDT0000 complete [ 6286.503387] LustreError: 217998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752211 with bad export cookie 17570287766188795075 [ 6286.506203] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6286.515763] LustreError: 217998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6286.754351] Lustre: server umount lustre-MDT0001 complete [ 6292.028403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6295.358993] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 6297.569190] LustreError: 222193:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6297.581588] LustreError: 222193:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 6300.512630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6303.101316] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 6307.911039] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6311.540503] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6316.089107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6320.158955] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 6324.673728] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6332.897408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6332.901908] LustreError: Skipped 1 previous similar message [ 6338.808570] Lustre: server umount lustre-OST0000 complete [ 6341.089408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6341.105034] Lustre: Skipped 5 previous similar messages [ 6347.339979] Lustre: server umount lustre-MDT0000 complete [ 6349.790430] LustreError: 224779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752274 with bad export cookie 17570287766188795957 [ 6349.792369] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6349.799758] LustreError: 224779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6350.018834] Lustre: server umount lustre-MDT0001 complete [ 6460.483607] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 6462.279239] Key type lgssc unregistered [ 6462.494550] LNet: 225851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6462.501742] LNetError: 225851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6462.513621] LNet: Removed LNI 192.168.206.141@tcp [ 6463.063260] Key type .llcrypt unregistered [ 6463.068766] Key type ._llcrypt unregistered [ 6479.198447] Key type ._llcrypt registered [ 6479.200873] Key type .llcrypt registered [ 6479.266362] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 6488.684724] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 6489.171571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6489.231490] alg: No test for adler32 (adler32-zlib) [ 6490.156735] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 6490.305623] LNet: Added LNI 192.168.206.141@tcp [8/256/0/180] [ 6491.919709] Key type lgssc registered [ 6492.624827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6514.020652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6514.029624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6515.154567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6515.178761] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6515.234222] Lustre: lustre-MDT0000: new disk, initializing [ 6515.269152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6515.283174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6517.278706] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 6523.269272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6523.347547] Lustre: 229175:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6523.385888] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6523.388709] Lustre: Skipped 1 previous similar message [ 6523.429528] Lustre: lustre-MDT0001: new disk, initializing [ 6523.477451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6523.505041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6523.516099] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6525.721257] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 6530.774357] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6533.808680] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6537.605797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6537.768677] Lustre: lustre-OST0000: new disk, initializing [ 6537.772892] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6537.777600] Lustre: 231142:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6537.826388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6541.369613] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 6546.955124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6546.967100] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6547.020163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6547.046505] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6549.425163] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6549.580741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6552.067391] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6552.182794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6557.153919] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6557.154984] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6557.157867] LustreError: Skipped 1 previous similar message [ 6557.159230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6557.163311] Lustre: Skipped 1 previous similar message [ 6559.407504] Lustre: server umount lustre-OST0000 complete [ 6562.273577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6562.280226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6562.282994] Lustre: Skipped 2 previous similar messages [ 6567.393243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6567.397637] Lustre: Skipped 1 previous similar message [ 6567.791619] Lustre: server umount lustre-MDT0000 complete [ 6569.939584] LustreError: 229167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752494 with bad export cookie 12451865124995473962 [ 6569.943495] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6569.949412] LustreError: 229167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6570.127531] Lustre: server umount lustre-MDT0001 complete [ 6580.747462] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 6585.942294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6586.170281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6588.646058] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 6591.458417] LustreError: 234072:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6591.482715] LustreError: 234072:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 6594.070884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6594.281086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6596.575173] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 6598.238554] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6602.082510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6602.290553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6605.428693] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 6609.415247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6609.501473] Lustre: lustre-OST0001: new disk, initializing [ 6609.504629] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6609.508520] Lustre: 236380:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6612.589448] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 6615.059776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6615.065189] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6615.130995] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6619.300994] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6622.945378] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6628.835829] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 23:42:32 (1788752552) [ 6630.369421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6630.372196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6630.375097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6630.380987] Lustre: Skipped 3 previous similar messages [ 6630.389838] Lustre: Skipped 3 previous similar messages [ 6631.989260] Lustre: server umount lustre-MDT0000 complete [ 6633.921882] LustreError: 234054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752558 with bad export cookie 12451865124995474844 [ 6633.929246] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6633.935657] LustreError: 234054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6634.136269] Lustre: server umount lustre-MDT0001 complete [ 6646.292383] Lustre: server umount lustre-OST0000 complete [ 6658.564198] Lustre: server umount lustre-OST0001 complete [ 6662.131712] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 6667.618234] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 6687.573910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6687.709562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6687.735782] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6687.788646] Lustre: lustre-MDT0000: new disk, initializing [ 6687.835868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6687.839047] Lustre: Skipped 1 previous similar message [ 6687.846416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6690.154038] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 6696.423112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6696.501375] Lustre: 240314:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6696.547373] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6696.550374] Lustre: Skipped 1 previous similar message [ 6696.598904] Lustre: lustre-MDT0001: new disk, initializing [ 6696.700770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6696.705320] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6698.939407] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 6704.620802] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6707.630584] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6711.376820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6711.551728] Lustre: 242283:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6713.561974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6713.567941] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6713.612514] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6715.385300] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 6720.764241] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6723.148372] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6723.266345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6725.402295] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6725.539827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6729.184261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6729.186897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6729.189920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6729.194994] Lustre: Skipped 2 previous similar messages [ 6729.202714] Lustre: Skipped 1 previous similar message [ 6732.680694] Lustre: server umount lustre-OST0000 complete [ 6737.890313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6737.899333] LustreError: Skipped 1 previous similar message [ 6737.900975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6737.908300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6740.802810] Lustre: server umount lustre-MDT0000 complete [ 6742.810623] LustreError: 241110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752667 with bad export cookie 12451865124995476965 [ 6742.815166] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6742.819100] LustreError: 241110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6742.968607] Lustre: server umount lustre-MDT0001 complete [ 6746.668824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6746.875640] LustreError: 244498:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6746.919339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6746.922017] Lustre: Skipped 2 previous similar messages [ 6748.807959] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 6752.226091] LustreError: 244499:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6752.882281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6755.247267] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 6759.458270] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6762.467476] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6766.120910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6769.557087] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 6773.532602] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6917.090326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6917.091585] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6917.095946] Lustre: Skipped 1 previous similar message [ 6917.099519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6917.104703] Lustre: Skipped 3 previous similar messages [ 6921.706180] Lustre: server umount lustre-OST0000 complete [ 6927.328446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6929.576350] Lustre: server umount lustre-MDT0000 complete [ 6931.211554] LustreError: 244480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752856 with bad export cookie 12451865124995477847 [ 6931.214170] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6931.216651] LustreError: 244480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6931.385281] Lustre: server umount lustre-MDT0001 complete [ 7037.470907] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 7039.019809] Key type lgssc unregistered [ 7039.172493] LNet: 248101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7039.178478] LNetError: 248101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7039.189083] LNet: Removed LNI 192.168.206.141@tcp [ 7039.622136] Key type .llcrypt unregistered [ 7039.624750] Key type ._llcrypt unregistered [ 7051.746598] Key type ._llcrypt registered [ 7051.748916] Key type .llcrypt registered [ 7051.800971] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 7059.961203] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 7060.421232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7060.435136] alg: No test for adler32 (adler32-zlib) [ 7061.331700] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 7061.476837] LNet: Added LNI 192.168.206.141@tcp [8/256/0/180] [ 7063.071799] Key type lgssc registered [ 7063.636385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7086.397330] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 7092.140794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7092.154979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7093.271849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7093.290624] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7093.352520] Lustre: lustre-MDT0000: new disk, initializing [ 7093.408868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7093.423831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7095.671146] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7102.289933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7102.356218] Lustre: 252143:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7102.399482] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7102.402367] Lustre: Skipped 1 previous similar message [ 7102.447294] Lustre: lustre-MDT0001: new disk, initializing [ 7102.506120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7102.546270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7102.556047] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7104.956065] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7108.222694] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7112.126195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7112.294786] Lustre: lustre-OST0000: new disk, initializing [ 7112.297451] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7112.300839] Lustre: 253771:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7112.350860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7115.674425] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7120.922108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7120.929314] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7120.985534] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7121.593376] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7121.691691] Lustre: lustre-OST0001: new disk, initializing [ 7121.694901] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7121.699356] Lustre: 254643:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7121.756530] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7124.650204] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7128.610224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7128.614569] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7128.681737] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7131.502779] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7136.061522] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7143.392264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7143.397044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7143.406297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7143.909239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7143.911412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7143.917049] Lustre: Skipped 1 previous similar message [ 7143.922394] Lustre: Skipped 2 previous similar messages [ 7146.941610] Lustre: server umount lustre-MDT0000 complete [ 7148.638463] LustreError: 254642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753073 with bad export cookie 11186480450988695824 [ 7148.641935] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7148.646222] LustreError: 254642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7148.809450] Lustre: server umount lustre-MDT0001 complete [ 7160.081991] Lustre: server umount lustre-OST0000 complete [ 7172.018880] Lustre: server umount lustre-OST0001 complete [ 7178.695314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7182.731111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7186.303672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7190.066649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7197.265672] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 7201.304668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7201.317347] Lustre: lustre-MDT0000: reset Object Index mappings [ 7201.541169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7203.288148] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7206.880647] LustreError: 259647:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7206.887269] LustreError: 259647:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 7207.284878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7207.303748] Lustre: lustre-MDT0001: reset Object Index mappings [ 7209.323820] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7210.782181] Lustre: 260789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7213.878607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7214.076268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7214.079443] Lustre: Skipped 1 previous similar message [ 7216.617089] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7217.129494] LustreError: 261577:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7219.956574] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7222.381595] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7226.345917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7226.346550] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 7231.543827] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7233.026931] Lustre: 262656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7236.580135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7236.583078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7236.585272] Lustre: Skipped 4 previous similar messages [ 7236.589424] Lustre: Skipped 2 previous similar messages [ 7240.755260] Lustre: server umount lustre-MDT0000 complete [ 7241.698352] LustreError: 259666:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7241.705767] LustreError: 259666:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 7242.236653] LustreError: 260791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753167 with bad export cookie 11186480450988702600 [ 7242.239945] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7242.241741] LustreError: 260791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7242.381757] Lustre: server umount lustre-MDT0001 complete [ 7254.030993] Lustre: server umount lustre-OST0000 complete [ 7265.894059] Lustre: server umount lustre-OST0001 complete [ 7273.631298] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 23:53:17 (1788753197) [ 7279.918243] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 7283.888642] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 7298.810381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7298.908947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7298.926272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7298.967296] Lustre: lustre-MDT0000: new disk, initializing [ 7298.996917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7298.999564] Lustre: Skipped 1 previous similar message [ 7299.004901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7300.561139] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7305.391087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7305.457849] Lustre: 266906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7305.608964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7307.287800] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7311.734075] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7313.812359] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7316.383453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7316.503879] Lustre: lustre-OST0000: new disk, initializing [ 7316.505902] Lustre: Skipped 1 previous similar message [ 7316.508140] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7316.510725] Lustre: Skipped 2 previous similar messages [ 7316.513471] Lustre: 268871:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7318.019207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7318.022830] Lustre: Skipped 1 previous similar message [ 7318.026330] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7318.065589] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7318.864464] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7322.929694] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7324.695314] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7324.795102] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7326.580464] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7326.685326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7328.225326] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7328.225821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7328.230086] LustreError: Skipped 1 previous similar message [ 7328.237494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7328.243141] Lustre: Skipped 1 previous similar message [ 7333.345490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7333.351992] Lustre: Skipped 1 previous similar message [ 7333.631779] Lustre: server umount lustre-OST0000 complete [ 7336.416626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7336.420573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7336.426260] Lustre: Skipped 1 previous similar message [ 7341.500354] Lustre: server umount lustre-MDT0000 complete [ 7343.200787] LustreError: 266898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753268 with bad export cookie 11186480450988704686 [ 7343.205988] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7343.207016] LustreError: 266898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7343.382994] Lustre: server umount lustre-MDT0001 complete [ 7351.352671] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 7355.974249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7356.163753] LustreError: 271802:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7356.181298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7356.185089] Lustre: Skipped 2 previous similar messages [ 7358.321672] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7361.511696] LustreError: 271803:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7362.963879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7365.072610] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7366.444220] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7369.582118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7372.691384] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7376.641261] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7376.742506] Lustre: lustre-OST0001: new disk, initializing [ 7376.744820] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7376.747583] Lustre: 274109:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7378.079729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7378.085827] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7378.120657] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7379.628641] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7384.990090] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7386.577204] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7388.741577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 7388.755152] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 7393.761992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7393.768432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7393.774405] Lustre: Skipped 1 previous similar message [ 7393.777513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7393.780556] Lustre: Skipped 3 previous similar messages [ 7395.058086] Lustre: server umount lustre-MDT0000 complete [ 7396.801509] LustreError: 271782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753321 with bad export cookie 11186480450988705568 [ 7396.813526] LustreError: 271782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7396.813623] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7396.837197] LustreError: 274577:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7396.846200] LustreError: 274577:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 7399.042391] Lustre: server umount lustre-MDT0001 complete [ 7403.476976] Lustre: server umount lustre-OST0000 complete [ 7405.650660] Lustre: server umount lustre-OST0001 complete [ 7414.237970] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 7419.532815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7419.735906] LustreError: 277448:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7421.931767] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7426.157607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7426.375934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7426.381417] Lustre: Skipped 4 previous similar messages [ 7428.414452] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7429.728866] Lustre: 278590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7432.919795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7435.921517] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7440.158256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7443.027499] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7444.401828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 7444.425562] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 7452.451194] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7454.048231] Lustre: 280467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7456.392250] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 7456.401966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 7461.786512] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 23:56:25 (1788753385) [ 7475.172141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7475.177323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7475.177980] Lustre: Skipped 6 previous similar messages [ 7475.181122] Lustre: Skipped 3 previous similar messages [ 7479.709484] Lustre: server umount lustre-MDT0000 complete [ 7480.289556] LustreError: 278179:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7480.297941] LustreError: 278179:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [ 7481.170932] LustreError: 277430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753406 with bad export cookie 11186480450988711938 [ 7481.174602] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7481.176083] LustreError: 277430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7487.349219] Lustre: server umount lustre-MDT0001 complete [ 7503.327350] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7503.378931] Lustre: server umount lustre-OST0000 complete [ 7519.199104] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7519.283853] Lustre: server umount lustre-OST0001 complete [ 7522.321322] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 7526.664540] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 7544.552789] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 7548.882513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7549.012863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7549.030334] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7549.075244] Lustre: lustre-MDT0000: new disk, initializing [ 7549.131352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7551.001376] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7556.237359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7556.289992] Lustre: 284992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7556.398626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7556.401067] Lustre: Skipped 3 previous similar messages [ 7556.416161] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7558.217560] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7564.161799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7564.292948] Lustre: 286629:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7565.493424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7566.970893] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7571.974031] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7574.083539] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7574.809184] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7579.907384] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7581.492818] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7581.496523] Lustre: Skipped 1 previous similar message [ 7587.121698] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 23:58:30 (1788753510) [ 7589.346664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7589.349390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7589.352574] Lustre: Skipped 6 previous similar messages [ 7589.354874] Lustre: Skipped 6 previous similar messages [ 7594.153153] Lustre: server umount lustre-MDT0000 complete [ 7594.464249] LustreError: 285003:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7594.472334] LustreError: 285003:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 7 previous similar messages [ 7595.691170] LustreError: 284985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753520 with bad export cookie 11186480450988719883 [ 7595.694818] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7595.697871] LustreError: 284985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7595.817686] Lustre: server umount lustre-MDT0001 complete [ 7607.827052] Lustre: server umount lustre-OST0000 complete [ 7619.780850] Lustre: server umount lustre-OST0001 complete [ 7623.822815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7625.932179] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7629.671944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7631.814203] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7634.917889] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7637.275492] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7640.089093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7642.709506] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7645.719229] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7662.089673] loop1: detected capacity change from 0 to 204800000 [ 7662.203070] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7662.207626] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7662.214895] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7664.538209] loop1: detected capacity change from 0 to 204800000 [ 7666.931294] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7666.970977] LustreError: 289729:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs23895: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 7666.978457] LustreError: 289729:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs23895 [ 7666.981783] LustreError: 289729:0:(mgs_handler.c:517:mgs_target_reg()) fs23895-MDT0001: Error checking target index: rc = -13 [ 7666.986878] LustreError: fs23895-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7666.992514] LustreError: 293478:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 7666.997052] LustreError: 293478:0:(tgt_mount.c:2128:server_put_super()) no obd fs23895-MDT0001 [ 7667.000291] LustreError: 293478:0:(tgt_mount.c:116:server_deregister_mount()) fs23895-MDT0001 not registered [ 7667.023115] Lustre: server umount fs23895-MDT0001 complete [ 7667.025121] LustreError: 293478:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7670.918333] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7670.971735] Lustre: Setting parameter fs23895-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs23895-MDT0001 [ 7670.998491] Lustre: srv-fs23895-MDT0001: No data found on store. Initialize space. [ 7671.000888] Lustre: Skipped 4 previous similar messages [ 7671.040073] Lustre: fs23895-MDT0001: new disk, initializing [ 7671.042318] Lustre: Skipped 3 previous similar messages [ 7671.090857] Lustre: srv-fs23895-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7671.096938] Lustre: 293944:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs23895-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7671.101742] LustreError: 293944:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs23895-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7673.128579] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7677.060951] Lustre: server umount fs23895-MDT0001 complete [ 7682.800351] loop1: detected capacity change from 0 to 204800000 [ 7685.096450] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7685.253406] Lustre: 295514:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs23895-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7685.258413] LustreError: 295514:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs23895-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7686.870397] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7689.002349] Lustre: server umount fs23895-MDT0001 complete [ 7693.793673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7693.795561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7693.798961] Lustre: Skipped 2 previous similar messages [ 7693.802266] Lustre: Skipped 2 previous similar messages [ 7699.442298] Lustre: server umount lustre-MDT0000 complete [ 7703.108322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7703.196934] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7705.278614] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7708.647810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7708.695259] loop1: detected capacity change from 0 to 204800000 [ 7711.301466] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7711.467936] Lustre: 297956:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs23895-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7711.472546] LustreError: 297956:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs23895-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7713.366915] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7715.624283] Lustre: server umount fs23895-MDT0001 complete [ 7722.872137] LustreError: 291570:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7722.883588] LustreError: 291570:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 44 previous similar messages [ 7726.562403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7730.714619] Lustre: server umount lustre-OST0000 complete [ 7738.502452] Lustre: server umount lustre-MDT0000 complete [ 7740.532476] LustreError: 290484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753665 with bad export cookie 11186480450988722557 [ 7740.540017] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7740.541515] LustreError: 290484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7740.710773] Lustre: server umount lustre-MDT0001 complete [ 7745.744206] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 7747.197188] Key type lgssc unregistered [ 7747.375954] LNet: 300309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7747.382846] LNetError: 300309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7747.393133] LNet: Removed LNI 192.168.206.141@tcp [ 7747.895173] Key type .llcrypt unregistered [ 7747.896923] Key type ._llcrypt unregistered [ 7757.488428] Key type ._llcrypt registered [ 7757.490193] Key type .llcrypt registered [ 7757.537184] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 7757.923678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7757.974652] alg: No test for adler32 (adler32-zlib) [ 7758.854832] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 7758.976425] LNet: Added LNI 192.168.206.141@tcp [8/256/0/180] [ 7760.583533] Key type lgssc registered [ 7761.136811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7764.996149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7765.008746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7766.263554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7768.172723] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7771.681878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7771.879058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7773.509139] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7776.471459] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7778.839845] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7781.797264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7781.957408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7784.320388] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7787.657034] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7791.206518] LustreError: 303383:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7791.214320] LustreError: 303383:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 7794.031050] LustreError: 303384:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7796.324247] LustreError: 303381:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7796.335713] LustreError: 303381:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 7799.160271] LustreError: 303384:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7804.282696] LustreError: 303381:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7804.289458] LustreError: 303381:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 7805.352581] loop1: detected capacity change from 0 to 204800000 [ 7805.434248] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7805.440519] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7805.447957] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7807.774805] loop1: detected capacity change from 0 to 204800000 [ 7809.934538] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7809.997333] Lustre: fs6036-OST0000: new disk, initializing [ 7809.999549] Lustre: srv-fs6036-OST0000: No data found on store. Initialize space. [ 7810.003103] Lustre: 305309:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7810.034656] Lustre: fs6036-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7812.293829] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7814.518764] LustreError: 304056:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7814.525340] LustreError: 304056:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [ 7815.916746] Lustre: server umount fs6036-OST0000 complete [ 7820.674098] loop1: detected capacity change from 0 to 204800000 [ 7822.739121] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7822.806542] Lustre: fs6036-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7825.088862] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7828.794501] loop2: detected capacity change from 0 to 204800000 [ 7828.883971] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7828.890861] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7828.898017] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7831.116377] loop2: detected capacity change from 0 to 204800000 [ 7832.549247] LustreError: 303381:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7832.558503] LustreError: 303381:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 10 previous similar messages [ 7833.667489] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7833.705683] LustreError: fs6036-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 7833.710572] LustreError: fs6036-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7833.715622] LustreError: 308344:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 7833.719252] LustreError: 308344:0:(tgt_mount.c:2128:server_put_super()) no obd fs6036-OST0001 [ 7833.721053] LustreError: 308344:0:(tgt_mount.c:116:server_deregister_mount()) fs6036-OST0001 not registered [ 7833.735473] Lustre: server umount fs6036-OST0001 complete [ 7833.737678] LustreError: 308344:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7838.365125] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7838.423301] Lustre: fs6036-OST0001: new disk, initializing [ 7838.425702] Lustre: srv-fs6036-OST0001: No data found on store. Initialize space. [ 7838.430450] Lustre: 308894:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7838.466879] Lustre: fs6036-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7841.082418] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7844.742073] Lustre: server umount fs6036-OST0000 complete [ 7848.187525] Lustre: server umount fs6036-OST0001 complete [ 7853.068856] loop1: detected capacity change from 0 to 204800000 [ 7855.600536] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7855.693016] Lustre: fs6036-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7858.384069] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7861.534353] loop2: detected capacity change from 0 to 204800000 [ 7863.983455] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7865.719033] LustreError: 304059:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7865.727568] LustreError: 304059:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 18 previous similar messages [ 7866.669039] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7870.424792] Lustre: server umount fs6036-OST0000 complete [ 7874.047075] Lustre: server umount fs6036-OST0001 complete [ 7878.627583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7878.628272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7878.632617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7878.634329] LustreError: Skipped 2 previous similar messages [ 7883.745381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7883.747732] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 7883.748458] Lustre: Skipped 2 previous similar messages [ 7884.321750] Lustre: server umount lustre-OST0000 complete [ 7888.865217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7888.866217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7888.870642] Lustre: Skipped 2 previous similar messages [ 7893.984475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7893.986764] Lustre: Skipped 2 previous similar messages [ 7899.104185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7899.106365] Lustre: Skipped 1 previous similar message [ 7900.127151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7900.202380] Lustre: server umount lustre-MDT0000 complete [ 7901.661840] LustreError: 301583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753826 with bad export cookie 16303642461243944032 [ 7901.664498] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7901.669897] LustreError: 301583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7901.806904] Lustre: server umount lustre-MDT0001 complete [ 7905.697044] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 7906.789148] Key type lgssc unregistered [ 7906.927860] LNet: 314805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7906.930968] LNetError: 314805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7906.942989] LNet: Removed LNI 192.168.206.141@tcp [ 7907.317178] Key type .llcrypt unregistered [ 7907.318809] Key type ._llcrypt unregistered [ 7919.710962] Key type ._llcrypt registered [ 7919.712230] Key type .llcrypt registered [ 7919.754788] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 7926.720469] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 7927.196433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7927.204067] alg: No test for adler32 (adler32-zlib) [ 7928.086034] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 7928.193496] LNet: Added LNI 192.168.206.141@tcp [8/256/0/180] [ 7929.783207] Key type lgssc registered [ 7930.230989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7945.087214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7945.096991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7946.202867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7946.215937] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7946.255099] Lustre: lustre-MDT0000: new disk, initializing [ 7946.286493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7946.296605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7947.908730] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7952.427608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7952.475833] Lustre: 318633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7952.508247] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7952.510756] Lustre: Skipped 1 previous similar message [ 7952.550446] Lustre: lustre-MDT0001: new disk, initializing [ 7952.593980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7952.609606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7952.614156] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7954.168970] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7958.295445] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7960.431546] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7963.130587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7963.242665] Lustre: lustre-OST0000: new disk, initializing [ 7963.245291] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7963.248895] Lustre: 320600:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7963.283481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7965.526384] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 7969.685874] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7972.296560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7972.300943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7972.345910] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7977.459917] Lustre: Mounted lustre-client - version 2.17.58_39_g34d8757 [ 7977.828294] Lustre: Unmounted lustre-client [ 7982.561497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7982.562118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7982.564653] LustreError: Skipped 1 previous similar message [ 7982.565493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7982.570621] Lustre: Skipped 1 previous similar message [ 7987.218282] Lustre: server umount lustre-OST0000 complete [ 7988.705127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7988.708777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7988.714707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7988.716902] Lustre: Skipped 1 previous similar message [ 7992.800741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7992.801078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7994.851451] Lustre: server umount lustre-MDT0000 complete [ 7996.294627] LustreError: 318625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753921 with bad export cookie 9130001094022384452 [ 7996.295804] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7996.298984] LustreError: 318625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7996.430679] Lustre: server umount lustre-MDT0001 complete [ 8000.493547] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 8001.729684] Key type lgssc unregistered [ 8021.343497] LNet: 322756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8021.347240] LNetError: 322756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8099.173367] LNet: Removed LNI 192.168.206.141@tcp [ 8099.518718] Key type .llcrypt unregistered [ 8099.520295] Key type ._llcrypt unregistered [ 8106.446354] Key type ._llcrypt registered [ 8106.448188] Key type .llcrypt registered [ 8106.493210] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 8113.854108] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 8114.280057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8114.341333] alg: No test for adler32 (adler32-zlib) [ 8115.223161] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 8115.342623] LNet: Added LNI 192.168.206.141@tcp [8/256/0/180] [ 8116.935190] Key type lgssc registered [ 8117.350723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8129.318773] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 00:07:33 (1788754053) [ 8132.087400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8132.093453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8133.184909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8133.200817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8133.240149] Lustre: lustre-MDT0000: new disk, initializing [ 8133.270577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8133.278904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8134.983196] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 8139.683752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8139.728561] Lustre: 325950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8139.752772] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8139.755331] Lustre: Skipped 1 previous similar message [ 8139.791968] Lustre: lustre-MDT0001: new disk, initializing [ 8139.827945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8139.842332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8139.846748] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8141.405693] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 8145.497616] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8147.622480] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8150.150696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8150.275620] Lustre: lustre-OST0000: new disk, initializing [ 8150.280198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8150.285805] Lustre: 327918:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8150.331074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8151.423197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8151.429037] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8151.471789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8152.622267] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 8157.079893] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8166.881208] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8166.884952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8166.887816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8166.890472] Lustre: Skipped 1 previous similar message [ 8166.894536] Lustre: Skipped 1 previous similar message [ 8169.638607] Lustre: server umount lustre-OST0000 complete [ 8172.000827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8172.001990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8172.006520] Lustre: Skipped 1 previous similar message [ 8172.012059] Lustre: Skipped 1 previous similar message [ 8177.121572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8177.124903] Lustre: Skipped 1 previous similar message [ 8177.312976] Lustre: server umount lustre-MDT0000 complete [ 8178.891387] LustreError: 326745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754103 with bad export cookie 4659932728883772915 [ 8178.895136] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8178.897380] LustreError: 326745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8179.031980] Lustre: server umount lustre-MDT0001 complete [ 8183.278448] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 8184.541612] Key type lgssc unregistered [ 8184.703558] LNet: 329821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8184.708442] LNetError: 329821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8184.717655] LNet: Removed LNI 192.168.206.141@tcp [ 8185.079483] Key type .llcrypt unregistered [ 8185.080945] Key type ._llcrypt unregistered [ 8194.156392] Key type ._llcrypt registered [ 8194.157892] Key type .llcrypt registered [ 8194.211238] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 8194.644170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8194.668980] alg: No test for adler32 (adler32-zlib) [ 8195.556342] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 8195.672562] LNet: Added LNI 192.168.206.141@tcp [8/256/0/180] [ 8197.263202] Key type lgssc registered [ 8197.758495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8201.008965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8201.016972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8202.198389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8203.786203] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 8207.234188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8207.389057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8209.029719] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 8211.892466] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8214.093428] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8216.602536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8216.738409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8219.002081] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 8222.066827] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8236.556400] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8237.121805] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8239.585580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8239.586458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8239.589979] Lustre: Skipped 2 previous similar messages [ 8239.591883] Lustre: Skipped 1 previous similar message [ 8243.069754] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.41@tcp (stopping) [ 8244.696220] Lustre: server umount lustre-MDT0000 complete [ 8244.706400] LustreError: 333558:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8244.712225] LustreError: 333558:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 8246.010362] LustreError: 334004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754171 with bad export cookie 5680087022448681204 [ 8246.013933] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8246.014621] LustreError: 334004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8248.181493] LustreError: 331062:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8248.187527] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.41@tcp (stopping) [ 8249.823983] LustreError: 331064:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8249.824465] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8251.872136] LustreError: 333058:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8251.873150] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8251.882059] Lustre: Skipped 1 previous similar message [ 8252.210737] Lustre: server umount lustre-MDT0001 complete [ 8259.756354] Lustre: server umount lustre-OST0000 complete [ 8263.220747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8264.802400] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 8265.901757] Lustre: 335266:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.141@tcp [ 8266.693882] Lustre: server umount MGS complete [ 8269.794268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8269.962184] LustreError: 335837:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8269.987515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8271.472677] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 8274.588872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8274.730523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8276.266665] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 8279.315430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8281.623103] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 8284.568548] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8285.140607] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 8287.321127] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8287.907128] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8289.760198] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8289.762381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8289.766877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8289.771150] Lustre: Skipped 1 previous similar message [ 8291.351967] Lustre: server umount lustre-OST0000 complete [ 8294.807860] Lustre: server umount lustre-MDT0000 complete [ 8294.881408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8294.882365] LustreError: 335832:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8294.886294] Lustre: Skipped 1 previous similar message [ 8296.185251] LustreError: 337291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754221 with bad export cookie 5680087022448682037 [ 8296.188263] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8296.189346] LustreError: 337291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8296.310406] Lustre: server umount lustre-MDT0001 complete [ 8300.113337] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 8301.254575] Key type lgssc unregistered [ 8301.398600] LNet: 339269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8301.402581] LNetError: 339269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8301.413533] LNet: Removed LNI 192.168.206.141@tcp [ 8301.768235] Key type .llcrypt unregistered [ 8301.769867] Key type ._llcrypt unregistered [ 8308.581887] Key type ._llcrypt registered [ 8308.583254] Key type .llcrypt registered [ 8308.627321] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 8315.859859] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 8316.307143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8316.315226] alg: No test for adler32 (adler32-zlib) [ 8317.182582] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 8317.283951] LNet: Added LNI 192.168.206.141@tcp [8/256/0/180] [ 8318.871194] Key type lgssc registered [ 8319.257088] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8333.148777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8333.157151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8334.254286] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8334.269556] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8334.324958] Lustre: lustre-MDT0000: new disk, initializing [ 8334.356232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8334.364232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8335.876939] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 8340.667641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8340.726478] Lustre: 342400:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8340.754181] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8340.755959] Lustre: Skipped 1 previous similar message [ 8340.792482] Lustre: lustre-MDT0001: new disk, initializing [ 8340.837325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8340.854015] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8340.859855] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8342.512711] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 8346.935224] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8349.329725] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8352.263512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8352.378764] Lustre: lustre-OST0000: new disk, initializing [ 8352.381211] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8352.384682] Lustre: 344367:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8352.420528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8354.714275] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 8359.072878] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8360.864551] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8362.496396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8362.500372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8362.544395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8362.996241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 8364.603338] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8364.691805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8367.585611] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8367.586253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8367.588898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8367.589521] LustreError: Skipped 1 previous similar message [ 8367.594508] Lustre: Skipped 1 previous similar message [ 8371.551142] Lustre: server umount lustre-OST0000 complete [ 8376.800232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8376.804898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8376.812603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8376.816445] Lustre: Skipped 1 previous similar message [ 8377.824555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8377.824802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8377.831662] Lustre: Skipped 1 previous similar message [ 8379.295250] Lustre: server umount lustre-MDT0000 complete [ 8380.980355] LustreError: 342437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754305 with bad export cookie 3547761533277739900 [ 8380.984167] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8380.985756] LustreError: 342437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8381.180888] Lustre: server umount lustre-MDT0001 complete [ 8384.632001] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 00:11:48 (1788754308) [ 8390.900748] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 8394.982055] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 8409.198140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8409.309406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8409.322325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8409.358148] Lustre: lustre-MDT0000: new disk, initializing [ 8409.391086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8409.399263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8410.936371] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 8415.544450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8415.595770] Lustre: 349132:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8415.626413] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8415.628929] Lustre: Skipped 1 previous similar message [ 8415.675757] Lustre: lustre-MDT0001: new disk, initializing [ 8415.718687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8415.736942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8415.741112] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8417.330160] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 8421.430534] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8423.722195] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8426.257436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8426.365158] Lustre: lustre-OST0000: new disk, initializing [ 8426.367134] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8426.370806] Lustre: 351100:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8426.399387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8426.515290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8426.519465] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8426.552776] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8428.617878] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 8432.798466] Lustre: DEBUG MARKER: oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8446.946256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8446.947424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8446.947441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8446.953034] Lustre: Skipped 1 previous similar message [ 8448.725125] Lustre: server umount lustre-OST0000 complete [ 8451.552322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8451.555635] LustreError: Skipped 1 previous similar message [ 8451.557201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8451.562220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8451.564712] Lustre: Skipped 1 previous similar message [ 8456.347966] Lustre: server umount lustre-MDT0000 complete [ 8457.787640] LustreError: 349928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754382 with bad export cookie 3547761533277740768 [ 8457.789550] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8457.792444] LustreError: 349928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8457.946496] Lustre: server umount lustre-MDT0001 complete [ 8462.083360] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 8463.217503] Key type lgssc unregistered [ 8463.348553] LNet: 353246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8463.351780] LNetError: 353246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8463.359445] LNet: Removed LNI 192.168.206.141@tcp [ 8463.719267] Key type .llcrypt unregistered [ 8463.720626] Key type ._llcrypt unregistered [ 8496.194704] Key type ._llcrypt registered [ 8496.196039] Key type .llcrypt registered [ 8496.241373] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 8496.369426] Key type .llcrypt unregistered [ 8496.371383] Key type ._llcrypt unregistered [ 8527.067936] Key type ._llcrypt registered [ 8527.069511] Key type .llcrypt registered [ 8527.126568] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 8527.254827] Key type .llcrypt unregistered [ 8527.255932] Key type ._llcrypt unregistered [ 8540.354963] Key type ._llcrypt registered [ 8540.356531] Key type .llcrypt registered [ 8540.411182] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 8540.539449] Key type .llcrypt unregistered [ 8540.541163] Key type ._llcrypt unregistered [ 8553.337786] Key type ._llcrypt registered [ 8553.339466] Key type .llcrypt registered [ 8553.389689] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 8553.521507] Key type .llcrypt unregistered [ 8553.522755] Key type ._llcrypt unregistered [ 8586.477349] Key type ._llcrypt registered [ 8586.478974] Key type .llcrypt registered [ 8586.523208] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 8586.642261] Key type .llcrypt unregistered [ 8586.643657] Key type ._llcrypt unregistered [ 8598.998942] Key type ._llcrypt registered [ 8599.000634] Key type .llcrypt registered [ 8599.054267] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 8599.205214] Key type .llcrypt unregistered [ 8599.206405] Key type ._llcrypt unregistered [ 8611.634098] Key type ._llcrypt registered [ 8611.635511] Key type .llcrypt registered [ 8611.681961] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 8611.803259] Key type .llcrypt unregistered [ 8611.804681] Key type ._llcrypt unregistered [ 8624.714723] Key type ._llcrypt registered [ 8624.716170] Key type .llcrypt registered [ 8624.761435] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 8624.887233] Key type .llcrypt unregistered [ 8624.888824] Key type ._llcrypt unregistered [ 8637.647930] Key type ._llcrypt registered [ 8637.648877] Key type .llcrypt registered [ 8637.695407] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 8637.832622] Key type .llcrypt unregistered [ 8637.834119] Key type ._llcrypt unregistered [ 8651.198374] Key type ._llcrypt registered [ 8651.199878] Key type .llcrypt registered [ 8651.247392] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 8651.366426] Key type .llcrypt unregistered [ 8651.368412] Key type ._llcrypt unregistered [ 8677.628117] Key type ._llcrypt registered [ 8677.629727] Key type .llcrypt registered [ 8677.679393] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 8677.828337] Key type .llcrypt unregistered [ 8677.830784] Key type ._llcrypt unregistered [ 8690.583791] Key type ._llcrypt registered [ 8690.584897] Key type .llcrypt registered [ 8690.623766] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 8690.739292] Key type .llcrypt unregistered [ 8690.740679] Key type ._llcrypt unregistered [ 8703.230785] Key type ._llcrypt registered [ 8703.232216] Key type .llcrypt registered [ 8703.271628] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 8703.393612] Key type .llcrypt unregistered [ 8703.395275] Key type ._llcrypt unregistered [ 8725.062483] Key type ._llcrypt registered [ 8725.063699] Key type .llcrypt registered [ 8725.113160] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 8732.034789] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 8732.376513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8732.396879] alg: No test for adler32 (adler32-zlib) [ 8733.286533] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 8733.398400] LNet: Added LNI 192.168.206.141@tcp [8/256/0/180] [ 8734.991458] Key type lgssc registered [ 8735.423155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8749.057914] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8749.067112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8750.174038] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 8750.202439] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 8750.242680] Lustre: tst250fs-MDT0000: new disk, initializing [ 8750.283257] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8750.292825] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8752.041956] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 8757.759438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8757.830818] Lustre: 373097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8757.869400] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 8757.872184] Lustre: Skipped 1 previous similar message [ 8757.912816] Lustre: tst250fs-MDT0001: new disk, initializing [ 8757.975622] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8757.996359] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8758.000940] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8759.873203] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 8762.485149] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8765.174549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8765.290801] Lustre: tst250fs-OST0000: new disk, initializing [ 8765.293116] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 8765.297139] Lustre: 374729:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8765.338068] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8767.537031] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 8770.051239] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8770.056246] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8770.108585] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8772.187557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8772.269721] Lustre: tst250fs-OST0001: new disk, initializing [ 8772.272220] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 8772.276220] Lustre: 375599:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8772.321093] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8774.845476] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_default_debug -1 all [ 8777.780539] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 8779.511136] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8779.516453] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8779.588628] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8789.987760] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8789.989517] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8789.993035] Lustre: Skipped 3 previous similar messages [ 8789.995854] Lustre: Skipped 2 previous similar messages [ 8795.104271] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8795.107433] Lustre: Skipped 2 previous similar messages [ 8800.225573] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8800.228732] Lustre: Skipped 4 previous similar messages [ 8804.319162] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8804.414497] Lustre: server umount tst250fs-MDT0000 complete [ 8806.134924] LustreError: 373088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754731 with bad export cookie 7405009081782046672 [ 8806.139218] LustreError: MGC192.168.206.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8806.142279] LustreError: 373088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8806.339300] Lustre: server umount tst250fs-MDT0001 complete [ 8818.697972] Lustre: server umount tst250fs-OST0000 complete [ 8830.638526] Lustre: server umount tst250fs-OST0001 complete [ 8838.171876] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 8841.837876] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 8855.540118] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 00:19:39 (1788754779) [ 8856.197522] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 8862.898895] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing set_hostid [ 8867.522968] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing load_modules_local [ 8881.025819] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8556 sec ========== 00:20:04 (1788754804) [ 8881.723994] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:20:05 (1788754805) === [ 8883.159474] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:20:06 (1788754806) === [ 8899.182779] Lustre: DEBUG MARKER: oleg641-server.virtnet: executing unload_modules_local [ 8900.369986] Key type lgssc unregistered [ 8900.517970] LNet: 384734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8900.521430] LNetError: 384734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8900.530577] LNet: Removed LNI 192.168.206.141@tcp [ 8900.927143] Key type .llcrypt unregistered [ 8900.928636] Key type ._llcrypt unregistered