[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 525261227 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002188] x2apic enabled [ 0.002961] Switched APIC routing to physical x2apic. [ 0.003008] kvm-guest: setup PV IPIs [ 0.005802] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007012] pid_max: default: 32768 minimum: 301 [ 0.008136] LSM: Security Framework initializing [ 0.009051] Yama: becoming mindful. [ 0.010040] SELinux: Initializing. [ 0.011072] *** VALIDATE selinux *** [ 0.019140] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023623] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024136] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026111] *** VALIDATE tmpfs *** [ 0.027417] *** VALIDATE proc *** [ 0.028230] *** VALIDATE cgroup *** [ 0.029009] *** VALIDATE cgroup2 *** [ 0.031006] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032150] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034032] Spectre V2 : User space: Vulnerable [ 0.035008] Speculative Store Bypass: Vulnerable [ 0.037787] debug: unmapping init [mem 0xffffffffad659000-0xffffffffad660fff] [ 0.039251] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040760] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041023] ... version: 2 [ 0.042016] ... bit width: 48 [ 0.043011] ... generic registers: 4 [ 0.044010] ... value mask: 0000ffffffffffff [ 0.045017] ... max period: 00007fffffffffff [ 0.046018] ... fixed-purpose events: 3 [ 0.047014] ... event mask: 000000070000000f [ 0.048340] rcu: Hierarchical SRCU implementation. [ 0.050479] smp: Bringing up secondary CPUs ... [ 0.051574] x86: Booting SMP configuration: [ 0.052027] .... node #0, CPUs: #1 #2 #3 [ 0.065228] smp: Brought up 1 node, 4 CPUs [ 0.067013] smpboot: Max logical packages: 1 [ 0.068021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.155803] node 0 deferred pages initialised in 85ms [ 0.158010] devtmpfs: initialized [ 0.159380] x86/mm: Memory block size: 128MB [ 0.162298] gcov: version magic: 0x41383552 [ 0.164349] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.165164] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.166463] pinctrl core: initialized pinctrl subsystem [ 0.167347] [ 0.167954] ************************************************************* [ 0.168024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169019] ** ** [ 0.170022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171025] ** ** [ 0.172021] ** This means that this kernel is built to expose internal ** [ 0.173026] ** IOMMU data structures, which may compromise security on ** [ 0.174022] ** your system. ** [ 0.175024] ** ** [ 0.176018] ** If you see this message and you are not debugging the ** [ 0.177022] ** kernel, report this immediately to your vendor! ** [ 0.178017] ** ** [ 0.179018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.180022] ************************************************************* [ 0.181847] NET: Registered protocol family 16 [ 0.182556] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.183085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.186013] cpuidle: using governor menu [ 0.187046] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.189640] PCI: Using configuration type 1 for base access [ 0.191141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.202085] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.203031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.205098] cryptd: max_cpu_qlen set to 1000 [ 0.207278] ACPI: Added _OSI(Module Device) [ 0.208019] ACPI: Added _OSI(Processor Device) [ 0.209018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.210021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.213967] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.216464] ACPI: Interpreter enabled [ 0.217092] ACPI: PM: (supports S0 S3 S4 S5) [ 0.218015] ACPI: Using IOAPIC for interrupt routing [ 0.219181] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.220322] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.230708] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.231113] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.232033] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.233141] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.235605] acpiphp: Slot [2] registered [ 0.236228] acpiphp: Slot [5] registered [ 0.237204] acpiphp: Slot [6] registered [ 0.238213] acpiphp: Slot [7] registered [ 0.239209] acpiphp: Slot [8] registered [ 0.240155] acpiphp: Slot [9] registered [ 0.241208] acpiphp: Slot [10] registered [ 0.242214] acpiphp: Slot [3] registered [ 0.243149] acpiphp: Slot [4] registered [ 0.244149] acpiphp: Slot [11] registered [ 0.245157] acpiphp: Slot [12] registered [ 0.246161] acpiphp: Slot [13] registered [ 0.247105] acpiphp: Slot [14] registered [ 0.248112] acpiphp: Slot [15] registered [ 0.249111] acpiphp: Slot [16] registered [ 0.250113] acpiphp: Slot [17] registered [ 0.251140] acpiphp: Slot [18] registered [ 0.252088] acpiphp: Slot [19] registered [ 0.253105] acpiphp: Slot [20] registered [ 0.254076] acpiphp: Slot [21] registered [ 0.255066] acpiphp: Slot [22] registered [ 0.256133] acpiphp: Slot [23] registered [ 0.257110] acpiphp: Slot [24] registered [ 0.258093] acpiphp: Slot [25] registered [ 0.259166] acpiphp: Slot [26] registered [ 0.260145] acpiphp: Slot [27] registered [ 0.261131] acpiphp: Slot [28] registered [ 0.262128] acpiphp: Slot [29] registered [ 0.263000] acpiphp: Slot [30] registered [ 0.263000] acpiphp: Slot [31] registered [ 0.265127] PCI host bridge to bus 0000:00 [ 0.267021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.269029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.272024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.274024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.277071] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.280033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.282242] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.285130] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.288365] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.298761] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.303914] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.307019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.309018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.311017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.314645] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.316785] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.319029] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.321759] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.325983] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.336016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.339030] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.343488] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.349016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.354038] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.371025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.383575] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.401017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.412024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.447037] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.465551] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.475040] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.483026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.503024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.517966] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.551027] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.570019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.607026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.614628] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.620022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.626022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.639023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.648280] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.655028] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.662026] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.678023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.688555] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.690442] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.693282] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.695408] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.698398] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.702208] iommu: Default domain type: Passthrough [ 0.703000] SCSI subsystem initialized [ 0.704345] ACPI: bus type USB registered [ 0.706420] usbcore: registered new interface driver usbfs [ 0.708179] usbcore: registered new interface driver hub [ 0.710122] usbcore: registered new device driver usb [ 0.711311] pps_core: LinuxPPS API ver. 1 registered [ 0.713016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.716103] PTP clock support registered [ 0.719117] EDAC MC: Ver: 3.0.0 [ 0.721147] PCI: Using ACPI for IRQ routing [ 0.723976] NetLabel: Initializing [ 0.725017] NetLabel: domain hash size = 128 [ 0.726014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.728114] NetLabel: unlabeled traffic allowed by default [ 0.731141] vgaarb: loaded [ 0.733192] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.736023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.743247] clocksource: Switched to clocksource kvm-clock [ 0.858579] VFS: Disk quotas dquot_6.6.0 [ 0.860392] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.863336] *** VALIDATE ramfs *** [ 0.864801] *** VALIDATE hugetlbfs *** [ 0.866655] pnp: PnP ACPI init [ 0.870362] pnp: PnP ACPI: found 6 devices [ 0.906812] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.910360] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.912298] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.914104] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.915910] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.918518] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.921411] NET: Registered protocol family 2 [ 0.923900] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.928853] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.932431] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.937191] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.940922] TCP: Hash tables configured (established 65536 bind 65536) [ 0.943974] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.946980] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.950228] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.953658] NET: Registered protocol family 1 [ 0.956386] RPC: Registered named UNIX socket transport module. [ 0.957897] RPC: Registered udp transport module. [ 0.958910] RPC: Registered tcp transport module. [ 0.960039] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.962050] NET: Registered protocol family 44 [ 0.963238] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.964941] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.966298] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.968170] PCI: CLS 0 bytes, default 64 [ 0.970118] Unpacking initramfs... [ 2.444582] debug: unmapping init [mem 0xffff9ce17cc54000-0xffff9ce17ffbffff] [ 2.451637] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.454097] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.458000] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.987688] Initialise system trusted keyrings [ 2.989559] Key type blacklist registered [ 2.992089] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.001823] zbud: loaded [ 3.004983] *** VALIDATE nfs *** [ 3.006403] *** VALIDATE nfs4 *** [ 3.008045] pstore: using deflate compression [ 3.013888] Platform Keyring initialized [ 3.125847] NET: Registered protocol family 38 [ 3.127854] Key type asymmetric registered [ 3.129507] Asymmetric key parser 'x509' registered [ 3.131672] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.135807] io scheduler mq-deadline registered [ 3.138582] io scheduler kyber registered [ 3.141162] io scheduler bfq registered [ 3.143569] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.148350] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.152472] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.156317] ACPI: Power Button [PWRF] [ 3.163092] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.181646] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.209204] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.219146] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.237436] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.271506] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.303529] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.314722] Non-volatile memory driver v1.3 [ 3.316625] Linux agpgart interface v0.103 [ 3.352068] virtio_blk virtio1: [vda] 145984 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.357125] vda: detected capacity change from 0 to 74743808 [ 3.393151] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.397130] vdb: detected capacity change from 0 to 1073741824 [ 3.412765] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.415455] vdc: detected capacity change from 0 to 2621440000 [ 3.431181] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.433087] vdd: detected capacity change from 0 to 2621440000 [ 3.447446] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.450434] vde: detected capacity change from 0 to 4294967296 [ 3.466829] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.469681] vdf: detected capacity change from 0 to 4294967296 [ 3.478558] libphy: Fixed MDIO Bus: probed [ 3.491261] usbcore: registered new interface driver usbserial_generic [ 3.493479] usbserial: USB Serial support registered for generic [ 3.495474] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.499508] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.501505] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.504326] mousedev: PS/2 mouse device common for all mice [ 3.508373] rtc_cmos 00:05: RTC can wake from S4 [ 3.513193] rtc_cmos 00:05: registered as rtc0 [ 3.514990] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.518098] intel_pstate: CPU model not supported [ 3.520597] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.524750] hid: raw HID events driver (C) Jiri Kosina [ 3.528424] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.530164] usbcore: registered new interface driver usbhid [ 3.536919] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.542641] usbhid: USB HID core driver [ 3.545309] drop_monitor: Initializing network drop monitor service [ 3.549157] Initializing XFRM netlink socket [ 3.552454] NET: Registered protocol family 10 [ 3.557365] Segment Routing with IPv6 [ 3.559240] NET: Registered protocol family 17 [ 3.561383] mpls_gso: MPLS GSO support [ 3.566125] RAS: Correctable Errors collector initialized. [ 3.568224] AVX version of gcm_enc/dec engaged. [ 3.569692] AES CTR mode by8 optimization enabled [ 3.655547] sched_clock: Marking stable (3655515771, 0)->(4628296553, -972780782) [ 3.659135] registered taskstats version 1 [ 3.661125] Loading compiled-in X.509 certificates [ 3.663423] zswap: loaded using pool lzo/zbud [ 3.692968] Key type big_key registered [ 3.707334] Key type encrypted registered [ 3.708682] ima: No TPM chip found, activating TPM-bypass! [ 3.710355] ima: Allocated hash algorithm: sha1 [ 3.711508] ima: No architecture policies found [ 3.713032] evm: Initialising EVM extended attributes: [ 3.714062] evm: security.selinux [ 3.715099] evm: security.ima [ 3.715750] evm: security.capability [ 3.716560] evm: HMAC attrs: 0x1 [ 3.719405] rtc_cmos 00:05: setting system clock to 2026-08-11 20:27:31 UTC (1786480051) [ 3.726961] debug: unmapping init [mem 0xffffffffae603000-0xffffffffae7fffff] [ 3.730064] debug: unmapping init [mem 0xffffffffad382000-0xffffffffad658fff] [ 3.743235] Write protecting the kernel read-only data: 28672k [ 3.747822] debug: unmapping init [mem 0xffffffffaba03000-0xffffffffabbfffff] [ 3.750642] debug: unmapping init [mem 0xffffffffac314000-0xffffffffac3fffff] [ 3.789226] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.798434] systemd[1]: Detected virtualization kvm. [ 3.800511] systemd[1]: Detected architecture x86-64. [ 3.801783] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.833491] systemd[1]: No hostname configured. [ 3.835419] systemd[1]: Set hostname to . [ 3.838766] random: systemd: uninitialized urandom read (16 bytes read) [ 3.842605] systemd[1]: Initializing machine ID from random generator. [ 3.936255] random: ln: uninitialized urandom read (6 bytes read) [ 4.040223] random: systemd: uninitialized urandom read (16 bytes read) [ 4.042848] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.047572] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.052150] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.775520] device-mapper: uevent: version 1.0.3 [ 4.778205] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 5.424389] random: fast init done [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.628165] virtio_net virtio0 ens2: renamed from eth0 [ 5.725536] scsi host0: ata_piix [ 5.757717] scsi host1: ata_piix [ 5.759484] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.761824] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.375583] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.221920] random: crng init done [ 10.222878] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.731323] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.058374] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.329883] SELinux: Disabled at runtime. [ 12.395558] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.403489] systemd[1]: Detected virtualization kvm. [ 12.405435] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.931884] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.936977] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.942323] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.948385] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.951831] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.962493] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.972035] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 13.009845] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.550883] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 14.038511] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.107473] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.253410] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.280251] EDAC sbridge: Ver: 1.1.2 [ 16.495229] Key type dns_resolver registered [ 17.315616] NFS: Registering the id_resolver key type [ 17.321187] Key type id_resolver registered [ 17.327414] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg138-server login: [ 48.455263] hrtimer: interrupt took 3306391 ns [ 93.581890] libcfs: loading out-of-tree module taints kernel. [ 93.675546] Key type ._llcrypt registered [ 93.677108] Key type .llcrypt registered [ 93.811204] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 115.756476] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 117.584670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 117.606813] alg: No test for adler32 (adler32-zlib) [ 119.389431] Lustre: Lustre: Build Version: 2.17.56_52_g54963d0 [ 120.583834] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 122.319195] Key type lgssc registered [ 124.213913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 145.644355] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 193.707897] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 207.971265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 208.012257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.374573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 209.400576] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 209.516362] Lustre: lustre-MDT0000: new disk, initializing [ 209.586379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.607189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 214.864723] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.330595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 230.470660] Lustre: 6508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 230.534629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 230.543929] Lustre: Skipped 1 previous similar message [ 230.723407] Lustre: lustre-MDT0001: new disk, initializing [ 230.865873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 230.903318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 230.924665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 236.152974] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 241.744910] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 252.835213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 253.227893] Lustre: lustre-OST0000: new disk, initializing [ 253.242814] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 253.254693] Lustre: 8445:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 253.343570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 255.388052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 255.395073] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 255.500138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 259.731811] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 277.265670] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 277.456776] Lustre: lustre-OST0001: new disk, initializing [ 277.460051] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 277.472362] Lustre: 9517:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 277.556825] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 284.182581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 284.191128] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 284.240514] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 285.956413] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 299.346661] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 308.248609] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 315.437704] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing check_logdir /tmp/testlogs/ [ 320.540713] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing yml_node [ 325.727732] Lustre: DEBUG MARKER: Client: 2.17.56.52 [ 328.246794] Lustre: DEBUG MARKER: MDS: 2.17.56.52 [ 330.882782] Lustre: DEBUG MARKER: OSS: 2.17.56.52 [ 332.512229] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Aug 11 16:32:58 EDT 2026 [ 351.329623] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 353.368935] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 371.169891] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 371.179725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 371.183709] Lustre: Skipped 3 previous similar messages [ 371.209830] Lustre: Skipped 3 previous similar messages [ 376.188281] Lustre: server umount lustre-MDT0000 complete [ 379.563513] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786480427 with bad export cookie 16519170666403582120 [ 379.570470] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 379.576584] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 381.409143] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 381.412724] LustreError: 6515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 381.413142] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 381.420068] Lustre: Skipped 1 previous similar message [ 381.438650] LustreError: 6515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 384.416664] LustreError: 10649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 384.418445] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 384.442719] LustreError: 10649:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 384.474798] Lustre: Skipped 2 previous similar messages [ 386.022083] Lustre: server umount lustre-MDT0001 complete [ 390.256586] Lustre: server umount lustre-OST0000 complete [ 393.980070] Lustre: server umount lustre-OST0001 complete [ 400.744809] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 410.175662] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 446.735122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 446.991749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 447.024762] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 447.106739] Lustre: lustre-MDT0000: new disk, initializing [ 447.199955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 447.222501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 451.396869] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 460.849860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 460.959979] Lustre: 15073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 461.010156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 461.012739] Lustre: Skipped 1 previous similar message [ 461.107602] Lustre: lustre-MDT0001: new disk, initializing [ 461.239957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 461.307074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 461.315706] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 465.344409] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 474.826105] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 480.808582] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 487.971900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 488.370861] Lustre: lustre-OST0000: new disk, initializing [ 488.373371] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 488.376941] Lustre: 17042:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 488.477588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 490.224063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 490.236261] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 490.387278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 495.072269] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 505.759988] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 511.180452] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 511.524151] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 517.264430] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 517.643305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 521.192860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 521.194276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 521.204751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 521.207151] LustreError: Skipped 1 previous similar message [ 521.238678] Lustre: Skipped 1 previous similar message [ 526.305314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 526.315379] Lustre: Skipped 1 previous similar message [ 533.983140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 534.133463] Lustre: server umount lustre-OST0000 complete [ 538.081739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 538.086585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 538.100784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 538.104616] Lustre: Skipped 3 previous similar messages [ 544.166196] Lustre: server umount lustre-MDT0000 complete [ 546.793518] LustreError: 15086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.819163] LustreError: 15086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 548.310112] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786480596 with bad export cookie 16519170666403583653 [ 548.314432] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 548.323186] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 548.711878] Lustre: server umount lustre-MDT0001 complete [ 554.062507] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 16:36:39 (1786480599) [ 570.045617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 570.617419] LustreError: 19917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.699650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 574.127606] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 575.970321] LustreError: 19918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 581.470924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.267988] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 592.109261] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 597.688853] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 600.050037] Lustre: Failing over lustre-MDT0000 [ 600.307577] Lustre: server umount lustre-MDT0000 complete [ 600.544778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 600.545850] LustreError: 19917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 600.552680] Lustre: Skipped 2 previous similar messages [ 600.565707] LustreError: 19917:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 616.935661] LustreError: 19918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 616.947650] LustreError: 19918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 619.567409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 619.722421] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 620.021346] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 625.122161] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 625.139276] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 625.176300] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 625.204469] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 633.875809] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 639.458103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 639.460069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 639.464741] Lustre: Skipped 1 previous similar message [ 639.483721] Lustre: Skipped 3 previous similar messages [ 644.539326] Lustre: server umount lustre-MDT0000 complete [ 648.096612] LustreError: 19897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786480695 with bad export cookie 16519170666403585067 [ 648.099754] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 648.107337] LustreError: 19897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 648.498893] Lustre: server umount lustre-MDT0001 complete [ 657.098413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 657.457550] LustreError: 23257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 657.476552] LustreError: 23257:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 657.546934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 657.549183] Lustre: Skipped 2 previous similar messages [ 661.946681] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 669.916941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 674.014709] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 681.114522] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 686.221957] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 688.111193] Lustre: Failing over lustre-MDT0000 [ 688.303297] Lustre: server umount lustre-MDT0000 complete [ 690.657329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 690.672985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 706.162518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 706.315515] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 706.698649] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 710.158958] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 711.650637] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 711.666448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 711.676283] Lustre: Skipped 1 previous similar message [ 711.694875] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 718.658964] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 727.010894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 727.013025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 727.022116] Lustre: Skipped 1 previous similar message [ 727.025099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 727.053230] Lustre: Skipped 1 previous similar message [ 728.615353] Lustre: server umount lustre-MDT0000 complete [ 731.844161] LustreError: 23237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786480779 with bad export cookie 16519170666403586005 [ 731.858715] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 732.099541] Lustre: server umount lustre-MDT0001 complete [ 740.003418] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 16:39:45 (1786480785) [ 753.630995] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 763.444547] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 800.232750] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 810.061924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 810.239387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 810.260427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 810.315672] Lustre: lustre-MDT0000: new disk, initializing [ 810.368572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 810.378752] Lustre: Skipped 2 previous similar messages [ 810.395356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 814.164402] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 823.563579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 823.718838] Lustre: 29625:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 824.000062] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 828.097200] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 835.981905] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 841.008414] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 846.561703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 846.825465] Lustre: 31596:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 848.636298] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 848.673786] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 848.756692] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 848.778449] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 852.610832] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 860.056460] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 866.363755] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 884.707616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 884.721603] Lustre: Skipped 1 previous similar message [ 884.722485] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 884.732262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 884.738224] LustreError: Skipped 1 previous similar message [ 884.752590] Lustre: Skipped 1 previous similar message [ 887.974539] Lustre: server umount lustre-OST0000 complete [ 897.908979] Lustre: server umount lustre-MDT0000 complete [ 900.066602] LustreError: 30493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 900.078656] LustreError: 30493:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 901.036223] LustreError: 29617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786480948 with bad export cookie 16519170666403586411 [ 901.047957] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 901.049457] LustreError: 29617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 901.405984] Lustre: server umount lustre-MDT0001 complete [ 911.148933] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 913.350363] Key type lgssc unregistered [ 913.616983] LNet: 33575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.623380] LNetError: 33575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.635168] LNet: Removed LNI 192.168.201.138@tcp [ 914.626158] Key type .llcrypt unregistered [ 914.631941] Key type ._llcrypt unregistered [ 938.334683] Key type ._llcrypt registered [ 938.340228] Key type .llcrypt registered [ 938.454692] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 939.445721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 939.627280] alg: No test for adler32 (adler32-zlib) [ 940.647737] Lustre: Lustre: Build Version: 2.17.56_52_g54963d0 [ 940.947147] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 942.599576] Key type lgssc registered [ 943.740915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 950.330749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 950.345109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 951.666688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 955.411904] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 962.030304] LustreError: 34989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 962.051041] LustreError: 34989:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 963.705987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 964.156708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 968.681746] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 975.480831] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 980.499601] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 987.081374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 987.389185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 988.403644] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 992.218559] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 998.211179] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1011.949069] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 16:44:17 (1786481057) [ 1013.618996] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1015.429817] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1021.460967] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 16:44:27 (1786481067) [ 1029.892826] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 16:44:35 (1786481075) [ 1030.985849] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1038.640285] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1038.644428] Lustre: Skipped 2 previous similar messages [ 1040.294746] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 16:44:45 (1786481085) [ 1043.238380] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1043.241302] Lustre: Skipped 3 previous similar messages [ 1060.200716] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1060.206582] Lustre: Skipped 6 previous similar messages [ 1062.519853] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 16:45:08 (1786481108) [ 1065.024214] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1068.295864] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1075.214922] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 16:45:21 (1786481121) [ 1076.659514] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1076.663597] Lustre: Skipped 3 previous similar messages [ 1086.208339] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 16:45:32 (1786481132) [ 1099.233313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1099.245511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1099.246990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1099.257674] Lustre: Skipped 1 previous similar message [ 1102.598722] Lustre: server umount lustre-OST0000 complete [ 1107.442419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1107.448181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1107.461380] Lustre: Skipped 1 previous similar message [ 1107.465963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1109.476196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1109.481900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1114.593837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1114.600559] Lustre: Skipped 2 previous similar messages [ 1119.713208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1120.235058] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1120.361349] Lustre: server umount lustre-MDT0000 complete [ 1123.362370] LustreError: 34970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786481171 with bad export cookie 16818094297142073802 [ 1123.375268] LustreError: 34970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1123.377419] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1123.651567] Lustre: server umount lustre-MDT0001 complete [ 1131.631993] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 1133.582605] Key type lgssc unregistered [ 1133.800820] LNet: 42848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1133.805632] LNetError: 42848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1133.816915] LNet: Removed LNI 192.168.201.138@tcp [ 1134.460214] Key type .llcrypt unregistered [ 1134.462259] Key type ._llcrypt unregistered [ 1150.112626] Key type ._llcrypt registered [ 1150.117833] Key type .llcrypt registered [ 1150.242592] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 1171.153237] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 1172.499281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1172.576096] alg: No test for adler32 (adler32-zlib) [ 1173.747623] Lustre: Lustre: Build Version: 2.17.56_52_g54963d0 [ 1173.987839] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 1175.703270] Key type lgssc registered [ 1177.400046] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1219.585137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1219.615681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1220.886212] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1220.915632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1221.001610] Lustre: lustre-MDT0000: new disk, initializing [ 1221.095621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1221.108455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1225.214319] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1235.900869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1236.061745] Lustre: 46003:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1236.131321] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1236.137130] Lustre: Skipped 1 previous similar message [ 1236.274628] Lustre: lustre-MDT0001: new disk, initializing [ 1236.399403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1236.443709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1236.452864] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1240.551996] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1250.080833] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1255.406969] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1261.404871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1261.636804] Lustre: lustre-OST0000: new disk, initializing [ 1261.639419] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1261.652344] Lustre: 47970:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1261.740478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1268.258096] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1270.812493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1270.821677] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1270.943835] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1277.247208] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1282.862065] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1283.194667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1288.059664] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1288.277231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1291.244421] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1291.254061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1291.268802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1291.749912] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1291.762941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1291.792773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1296.407592] Lustre: server umount lustre-OST0000 complete [ 1301.984598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1301.987737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1302.007200] Lustre: Skipped 1 previous similar message [ 1302.026877] Lustre: Skipped 1 previous similar message [ 1307.063534] Lustre: server umount lustre-MDT0000 complete [ 1311.015392] LustreError: 45996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786481358 with bad export cookie 13988991253348629917 [ 1311.017863] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1311.030936] LustreError: 45996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1311.370659] Lustre: server umount lustre-MDT0001 complete [ 1320.933908] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 16:49:26 (1786481366) [ 1327.745748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1328.166757] LustreError: 50431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.184126] LustreError: 50431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1328.246490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1333.029597] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1333.735520] LustreError: 50432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1337.826643] LustreError: 50431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1342.421155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1342.859027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1348.018321] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1355.810996] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1362.136914] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1369.969638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1370.443346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1377.158251] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1385.859344] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1393.974213] LustreError: 52959:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1395.065714] Lustre: Modifying parameter general.timeout=1 in log params [ 1396.160146] Lustre: Modifying parameter general.timeout=2 in log params [ 1399.000605] Lustre: Modifying parameter general.timeout=5 in log params [ 1399.003704] Lustre: Skipped 2 previous similar messages [ 1404.118549] Lustre: Modifying parameter general.timeout=10 in log params [ 1404.121113] Lustre: Skipped 4 previous similar messages [ 1412.979772] Lustre: Modifying parameter general.timeout=19 in log params [ 1412.985273] Lustre: Skipped 8 previous similar messages [ 1429.246911] Lustre: Modifying parameter general.timeout=38 in log params [ 1429.250558] Lustre: Skipped 18 previous similar messages [ 1461.949672] Lustre: Modifying parameter general.timeout=75 in log params [ 1461.952673] Lustre: Skipped 36 previous similar messages [ 1524.976609] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 16:52:50 (1786481570) [ 1526.042763] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1526.051345] Lustre: Skipped 55 previous similar messages [ 1527.810465] Lustre: 59912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1529.584232] Lustre: 60012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1533.557851] Lustre: 60209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1533.576737] Lustre: 60209:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1569.102751] Lustre: 61781:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1569.117959] Lustre: 61781:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1569.151261] LustreError: 61781:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1573.880818] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 16:53:39 (1786481619) [ 1588.708522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1588.721773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1588.742118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1593.315737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1593.320334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1593.335399] Lustre: Skipped 1 previous similar message [ 1593.348639] Lustre: Skipped 2 previous similar messages [ 1594.782308] Lustre: server umount lustre-MDT0000 complete [ 1598.437091] LustreError: 51953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1598.453722] LustreError: 51953:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1598.492735] LustreError: 50411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786481646 with bad export cookie 13988991253348630799 [ 1598.497591] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1598.505813] LustreError: 50411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1598.740511] Lustre: server umount lustre-MDT0001 complete [ 1613.449303] Lustre: server umount lustre-OST0000 complete [ 1640.689736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1640.979796] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1641.163693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1645.664697] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1653.934836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1654.066242] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1654.155353] Lustre: 64315:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1658.695436] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1665.782651] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1671.156746] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1678.523771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1678.755421] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1679.045927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1679.058226] Lustre: Skipped 1 previous similar message [ 1686.571369] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1694.823919] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1697.411224] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1697.422872] Lustre: Skipped 22 previous similar messages [ 1697.499276] Lustre: 66880:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1706.977423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1706.985121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1707.003882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1712.020149] Lustre: server umount lustre-OST0000 complete [ 1722.391619] Lustre: server umount lustre-MDT0000 complete [ 1722.848822] LustreError: 65389:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1722.864348] LustreError: 65389:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1725.848646] LustreError: 64306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786481773 with bad export cookie 13988991253348633893 [ 1725.865940] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1725.869388] LustreError: 64306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1726.138462] Lustre: server umount lustre-MDT0001 complete [ 1737.332300] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 1740.223645] Key type lgssc unregistered [ 1740.481560] LNet: 68094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1740.501250] LNetError: 68094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1740.514511] LNet: Removed LNI 192.168.201.138@tcp [ 1741.371457] Key type .llcrypt unregistered [ 1741.372625] Key type ._llcrypt unregistered [ 1766.376635] Key type ._llcrypt registered [ 1766.378177] Key type .llcrypt registered [ 1766.546707] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 1767.960296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1768.057027] alg: No test for adler32 (adler32-zlib) [ 1769.140749] Lustre: Lustre: Build Version: 2.17.56_52_g54963d0 [ 1769.374603] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 1771.023177] Key type lgssc registered [ 1771.888348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1778.932306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1778.958962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1780.450864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1784.530816] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1790.945709] LustreError: 69517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1790.970580] LustreError: 69517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1792.656832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1793.470693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1797.715806] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1804.983682] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1811.110962] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1817.795806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1818.299195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1823.606425] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1830.355604] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1838.325943] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1846.244636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1846.246789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1846.255656] Lustre: Skipped 1 previous similar message [ 1846.263914] Lustre: Skipped 2 previous similar messages [ 1851.365768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1851.378605] Lustre: Skipped 2 previous similar messages [ 1851.683239] Lustre: server umount lustre-MDT0000 complete [ 1854.718708] LustreError: 69496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786481902 with bad export cookie 1204255194428180767 [ 1854.726664] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1854.727369] LustreError: 69496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1854.942465] Lustre: server umount lustre-MDT0001 complete [ 1867.462356] Lustre: server umount lustre-OST0000 complete [ 1886.702557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1886.889567] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1886.908277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1887.072380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.033311] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1896.359730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1896.451840] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1896.478652] Lustre: 74387:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1896.860068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1900.837342] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1907.181955] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1911.527357] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1916.903682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1917.043586] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1917.322868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1922.341896] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1928.599747] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1930.063638] Lustre: 76964:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1931.766277] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1936.865582] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1936.879760] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1936.893580] Lustre: Skipped 1 previous similar message [ 1936.899687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1941.754810] Lustre: server umount lustre-OST0000 complete [ 1947.617026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1947.625689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1947.634295] Lustre: Skipped 2 previous similar messages [ 1947.649017] Lustre: Skipped 3 previous similar messages [ 1952.736140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1959.394897] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1959.564765] Lustre: server umount lustre-MDT0000 complete [ 1962.593686] LustreError: 76966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786482010 with bad export cookie 1204255194428181607 [ 1962.598260] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1962.600637] LustreError: 76966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1963.067341] Lustre: server umount lustre-MDT0001 complete [ 1972.118285] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 1973.965969] Key type lgssc unregistered [ 1974.180635] LNet: 78175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1974.183849] LNetError: 78175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1974.196868] LNet: Removed LNI 192.168.201.138@tcp [ 1974.731636] Key type .llcrypt unregistered [ 1974.736173] Key type ._llcrypt unregistered [ 1994.782546] Key type ._llcrypt registered [ 1994.787076] Key type .llcrypt registered [ 1994.852720] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 1995.856404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1995.904122] alg: No test for adler32 (adler32-zlib) [ 1996.852336] Lustre: Lustre: Build Version: 2.17.56_52_g54963d0 [ 1997.026894] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 1998.663165] Key type lgssc registered [ 1999.464419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2004.656477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2004.668824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2005.940462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2009.104397] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2015.929284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2016.336447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2020.500746] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2026.992956] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2031.633990] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2036.837541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2037.207236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2041.937854] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2047.446793] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2057.643062] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2058.340284] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2059.545486] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2059.551091] Lustre: Skipped 1 previous similar message [ 2062.141944] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2062.146229] Lustre: Skipped 3 previous similar messages [ 2066.757689] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2066.761882] Lustre: Skipped 6 previous similar messages [ 2075.093191] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2075.095956] Lustre: Skipped 11 previous similar messages [ 2091.237193] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2091.239657] Lustre: Skipped 25 previous similar messages [ 2128.123399] Lustre: 87687:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2128.794830] Lustre: Disabling parameter general.jobid_var= in log params [ 2128.798103] Lustre: Skipped 47 previous similar messages [ 2130.342976] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 17:02:56 (1786482176) [ 2131.668669] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2133.049307] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 17:02:58 (1786482178) [ 2137.575258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2137.580834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2137.595197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2139.619466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2139.620498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2139.622584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2144.745378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2144.751391] Lustre: Skipped 2 previous similar messages [ 2149.863018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2149.866065] Lustre: Skipped 1 previous similar message [ 2150.367115] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2150.460768] Lustre: server umount lustre-OST0000 complete [ 2154.469416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2154.485075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2154.495970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2159.519410] Lustre: server umount lustre-MDT0000 complete [ 2160.104229] LustreError: 79590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2160.118392] LustreError: 79590:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2162.098814] LustreError: 82190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786482209 with bad export cookie 8220317482528194792 [ 2162.116539] LustreError: 82190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2162.118695] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2162.425105] Lustre: server umount lustre-MDT0001 complete [ 2170.476474] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 2172.210128] Key type lgssc unregistered [ 2172.465583] LNet: 88939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2172.469656] LNetError: 88939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2172.482539] LNet: Removed LNI 192.168.201.138@tcp [ 2173.027322] Key type .llcrypt unregistered [ 2173.028887] Key type ._llcrypt unregistered [ 2175.906579] Key type ._llcrypt registered [ 2175.908437] Key type .llcrypt registered [ 2176.004717] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2179.097422] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules [ 2179.923625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2179.933563] alg: No test for adler32 (adler32-zlib) [ 2180.927971] Lustre: Lustre: Build Version: 2.17.56_52_g54963d0 [ 2181.092288] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 2182.711611] Key type lgssc registered [ 2183.459380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2184.329686] LustreError: 90140:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2199.260178] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2205.307845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2206.307258] LustreError: 90140:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2206.327103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2207.676273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2210.272281] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2215.659473] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 17:04:21 (1786482261) [ 2217.953401] LustreError: 91162:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2217.968809] LustreError: 91162:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2218.524114] LustreError: 91194:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2218.527535] LustreError: 91194:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2218.531690] LustreError: 91194:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 11, retries 0, failed: rc = -5 [ 2218.740196] Lustre: server umount lustre-MDT0000 complete [ 2227.305676] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 2228.771311] Key type lgssc unregistered [ 2228.933808] LNet: 92688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2228.937107] LNetError: 92688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2228.949337] LNet: Removed LNI 192.168.201.138@tcp [ 2229.400266] Key type .llcrypt unregistered [ 2229.404054] Key type ._llcrypt unregistered [ 2241.679096] Key type ._llcrypt registered [ 2241.680388] Key type .llcrypt registered [ 2241.737874] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2242.349364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2242.377197] alg: No test for adler32 (adler32-zlib) [ 2243.299651] Lustre: Lustre: Build Version: 2.17.56_52_g54963d0 [ 2243.468554] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 2245.071206] Key type lgssc registered [ 2245.734794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2250.261394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2250.268976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2251.575727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2254.322370] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2259.600885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.000904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2262.603529] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2266.807497] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2270.463725] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2274.614804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2274.939409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2278.478404] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2282.846464] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2315.611160] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 17:06:01 (1786482361) [ 2316.574712] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2317.709804] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 17:06:03 (1786482363) [ 2321.377364] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2321.379797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2321.385616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2321.391069] Lustre: Skipped 2 previous similar messages [ 2325.713302] Lustre: server umount lustre-MDT0000 complete [ 2326.503562] LustreError: 96672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2326.514655] LustreError: 96672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2327.817946] LustreError: 93864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786482375 with bad export cookie 15772077453022849439 [ 2327.825397] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2327.827751] LustreError: 93864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2327.983479] Lustre: server umount lustre-MDT0001 complete [ 2339.487430] Lustre: server umount lustre-OST0000 complete [ 2346.444708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2346.732169] LustreError: 98158:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2346.840073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2349.332591] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2352.096719] LustreError: 98159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2354.178057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2354.664260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2357.021865] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2361.927742] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2365.813149] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2374.640736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2374.757594] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2374.762600] LustreError: 98148:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2374.781178] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2374.789856] LustreError: 100239:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2374.793828] LustreError: 100239:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2374.799433] LustreError: 100239:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2374.844788] Lustre: server umount lustre-OST0000 complete [ 2374.847234] LustreError: 100239:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2381.108598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2381.293550] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2381.298780] LustreError: 98147:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2381.309122] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2381.321154] LustreError: 100701:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2381.327928] LustreError: 100701:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2381.332756] LustreError: 100701:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2381.383030] Lustre: server umount lustre-OST0000 complete [ 2381.385943] LustreError: 100701:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2387.796085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2387.979280] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2387.983596] Lustre: Found index 0 for lustre-OST0000, updating log [ 2387.990542] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2388.057774] Lustre: lustre-OST0000: new disk, initializing [ 2388.061107] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2388.067791] Lustre: 101238:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2388.272988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2392.170237] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2394.517183] Lustre: Failing over lustre-OST0000 [ 2394.610510] Lustre: server umount lustre-OST0000 complete [ 2396.641074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2396.642318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2396.652143] Lustre: Skipped 3 previous similar messages [ 2396.655863] Lustre: Skipped 1 previous similar message [ 2399.715628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2399.720474] Lustre: Skipped 1 previous similar message [ 2404.832787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2404.838467] Lustre: Skipped 1 previous similar message [ 2406.943848] Lustre: server umount lustre-MDT0000 complete [ 2409.296216] LustreError: 98139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786482457 with bad export cookie 15772077453022852176 [ 2409.299109] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2409.303635] LustreError: 98139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2409.516787] Lustre: server umount lustre-MDT0001 complete [ 2415.022241] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 17:07:41 (1786482461) [ 2426.738283] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2433.576078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2433.814257] LustreError: 103581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.894834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2436.937861] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2439.139128] LustreError: 103582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2441.735222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2444.552405] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2446.433019] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2450.616670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2451.193733] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2452.197610] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2452.204697] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2452.213316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2454.737593] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2457.571061] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2457.582163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2458.599158] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 2458.617757] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2458.619660] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2458.629423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2459.759825] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2459.807692] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2459.851988] Lustre: lustre-OST0001: new disk, initializing [ 2459.854818] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2459.860671] Lustre: 105899:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2462.954754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2462.959157] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2462.972161] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2463.027872] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2467.733303] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2471.218506] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2473.444267] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2473.445280] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2473.449876] Lustre: Skipped 2 previous similar messages [ 2473.452183] Lustre: Skipped 1 previous similar message [ 2479.262343] Lustre: server umount lustre-MDT0001 complete [ 2479.930990] LustreError: 105094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2484.108714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2484.162883] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2484.165799] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2484.198813] Lustre: 103569:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2484.570814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2484.577448] Lustre: Skipped 3 previous similar messages [ 2485.481815] LustreError: 107483:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2485.490078] LustreError: 107483:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2485.497606] LustreError: 103573:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2485.550940] LustreError: 107495:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.138@tcp: cfg command failed: rc = -17 [ 2485.557841] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2485.557841] [ 2486.812162] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2489.837794] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2492.032956] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 17:08:58 (1786482538) [ 2500.867766] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2510.165743] Lustre: 109503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2522.140265] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2523.933116] Lustre: 110638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2587.106385] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2587.109229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2587.111301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2587.111311] LustreError: Skipped 1 previous similar message [ 2587.112633] Lustre: Skipped 3 previous similar messages [ 2587.133222] Lustre: Skipped 8 previous similar messages [ 2590.734549] Lustre: server umount lustre-MDT0000 complete [ 2592.224762] LustreError: 103577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2592.233670] LustreError: 103577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2592.612841] LustreError: 103562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786482640 with bad export cookie 15772077453022853114 [ 2592.614577] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2592.618617] LustreError: 103562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2592.759648] Lustre: server umount lustre-MDT0001 complete [ 2604.577649] Lustre: server umount lustre-OST0000 complete [ 2616.966456] Lustre: server umount lustre-OST0001 complete [ 2620.034273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2625.584158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2630.360412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2635.293519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2644.994777] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2651.062125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2651.087267] Lustre: lustre-MDT0000: reset Object Index mappings [ 2651.347414] LustreError: 114586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2651.415873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2653.913345] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2658.882463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2658.894619] Lustre: lustre-MDT0001: reset Object Index mappings [ 2661.815717] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2663.384618] Lustre: 115734:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2667.134840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2668.663330] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 2671.224877] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2675.960744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2677.362091] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2677.365308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:382 to 0x280000401:418) [ 2677.431910] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2679.194494] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2683.858445] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2685.764466] Lustre: 117609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2711.713501] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 17:12:37 (1786482757) [ 2741.216595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2741.222951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.236145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2746.045544] Lustre: server umount lustre-MDT0000 complete [ 2747.809904] LustreError: 114620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786482795 with bad export cookie 15772077453022900168 [ 2747.810664] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2747.816154] LustreError: 114620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2749.408445] LustreError: 115542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2749.409175] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2749.414415] LustreError: 115542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2749.420817] Lustre: Skipped 4 previous similar messages [ 2754.211710] Lustre: server umount lustre-MDT0001 complete [ 2770.400052] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2770.463841] Lustre: server umount lustre-OST0000 complete [ 2778.428976] Lustre: server umount lustre-OST0001 complete [ 2781.666698] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 2785.581162] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2800.714310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2800.816858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2800.829872] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2800.869085] Lustre: lustre-MDT0000: new disk, initializing [ 2800.910866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2800.915491] Lustre: Skipped 3 previous similar messages [ 2800.923642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2802.660181] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2806.116803] Lustre: server umount lustre-MDT0000 complete [ 2809.904052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2810.026299] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2810.033191] Lustre: 121874:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2810.041813] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2811.801597] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2813.842925] Lustre: server umount lustre-MDT0000 complete [ 2817.270570] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 17:14:23 (1786482863) [ 2818.103274] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2818.822301] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 17:14:25 (1786482865) [ 2825.335049] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2829.612871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2831.575115] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2835.397450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2835.458490] Lustre: 123920:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2835.495076] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2835.497974] Lustre: Skipped 1 previous similar message [ 2835.541702] Lustre: lustre-MDT0001: new disk, initializing [ 2835.594850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2835.600454] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2837.448719] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2840.070696] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2842.849729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2843.016256] Lustre: lustre-OST0000: new disk, initializing [ 2843.018861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2843.021889] Lustre: 125500:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2845.067687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2845.072072] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2845.122420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2845.712312] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2851.163461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2851.262859] Lustre: lustre-OST0001: new disk, initializing [ 2851.265448] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2851.268414] Lustre: 126370:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2852.635038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2852.640746] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2852.693032] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2853.932184] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2859.167878] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2860.770987] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3011.016109] Lustre: 132750:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3011.019332] Lustre: 132750:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3011.022374] Lustre: 132750:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3011.025508] Lustre: 132750:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3011.028272] Lustre: 132750:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3011.031322] Lustre: 132750:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3099.010381] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 17:19:05 (1786483145) [ 3112.416095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3112.419220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3112.424752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3112.426711] Lustre: Skipped 8 previous similar messages [ 3114.083669] Lustre: server umount lustre-MDT0000 complete [ 3115.321361] LustreError: 123914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786483163 with bad export cookie 15772077453022960025 [ 3115.323851] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3115.325666] LustreError: 123914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3119.073642] LustreError: 127172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3119.079226] LustreError: 127172:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3121.561695] Lustre: server umount lustre-MDT0001 complete [ 3128.891584] Lustre: server umount lustre-OST0000 complete [ 3136.213982] Lustre: server umount lustre-OST0001 complete [ 3138.293392] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 3141.886547] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 3154.450227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3154.534494] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3154.547421] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3154.583850] Lustre: lustre-MDT0000: new disk, initializing [ 3154.614907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3154.617331] Lustre: Skipped 5 previous similar messages [ 3154.623568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3156.058415] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3160.456850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3160.500941] Lustre: 135761:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3160.607614] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3162.208229] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3166.240564] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3168.266840] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3170.583016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3170.693908] Lustre: 137728:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3172.410333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3172.412620] Lustre: Skipped 1 previous similar message [ 3172.414931] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3172.452202] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3172.844667] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3176.697029] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3180.033825] Lustre: lustre-MDD0000: changelog on [ 3180.906760] Lustre: lustre-MDD0001: changelog on [ 3497.960621] Lustre: lustre-MDD0001: changelog off [ 3498.568647] Lustre: 135769:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 3498.877638] Lustre: lustre-MDD0000: changelog off [ 3505.122413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3505.122511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3505.126301] Lustre: Skipped 5 previous similar messages [ 3505.128397] LustreError: Skipped 1 previous similar message [ 3505.129222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3505.133125] Lustre: Skipped 9 previous similar messages [ 3506.521771] Lustre: server umount lustre-OST0000 complete [ 3522.015078] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3522.123750] Lustre: server umount lustre-MDT0000 complete [ 3523.433322] LustreError: 135755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786483571 with bad export cookie 15772077453023761112 [ 3523.436286] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3523.437678] LustreError: 135755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3523.566279] Lustre: server umount lustre-MDT0001 complete [ 3527.351025] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 3528.364408] Key type lgssc unregistered [ 3528.490351] LNet: 145963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3528.493317] LNetError: 145963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3528.499340] LNet: Removed LNI 192.168.201.138@tcp [ 3528.796132] Key type .llcrypt unregistered [ 3528.797368] Key type ._llcrypt unregistered [ 3535.443698] Key type ._llcrypt registered [ 3535.444718] Key type .llcrypt registered [ 3535.483492] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 3541.978799] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 3542.450606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3542.463383] alg: No test for adler32 (adler32-zlib) [ 3543.324761] Lustre: Lustre: Build Version: 2.17.56_52_g54963d0 [ 3543.402649] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 3544.983171] Key type lgssc registered [ 3545.313983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3555.755830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3555.760600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3556.835639] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3556.845345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3556.882256] Lustre: lustre-MDT0000: new disk, initializing [ 3556.901223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3556.908033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3558.168076] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3562.179392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3562.219951] Lustre: 149134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3562.243858] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3562.246157] Lustre: Skipped 1 previous similar message [ 3562.277868] Lustre: lustre-MDT0001: new disk, initializing [ 3562.309518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3562.321204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3562.324298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3563.564545] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3567.009678] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3568.708679] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3570.890582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3570.982623] Lustre: lustre-OST0000: new disk, initializing [ 3570.985101] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3570.987377] Lustre: 151103:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3571.014146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3571.762898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3571.766680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3571.797841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3572.815184] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3576.355533] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3578.493123] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_module obdecho/obdecho [ 3579.063990] LustreError: 152105:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3579.067732] LustreError: 152105:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 3579.605732] LustreError: 152201:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 3582.943934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3582.946558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3582.951904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3587.040694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3587.043088] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3588.629753] Lustre: server umount lustre-MDT0000 complete [ 3589.749172] LustreError: 149126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786483637 with bad export cookie 15966460135711628498 [ 3589.751887] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3589.753664] LustreError: 149126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3589.856665] Lustre: server umount lustre-MDT0001 complete [ 3601.235135] Lustre: server umount lustre-OST0000 complete [ 3607.645164] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 3610.801046] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 3621.610725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3621.679119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3621.688450] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3621.721920] Lustre: lustre-MDT0000: new disk, initializing [ 3621.741602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3621.746904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3622.987205] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3626.785659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3626.823458] Lustre: 155909:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3626.846275] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3626.848602] Lustre: Skipped 1 previous similar message [ 3626.882166] Lustre: lustre-MDT0001: new disk, initializing [ 3626.912619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3626.922760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3626.926066] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3628.110685] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3631.738502] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3633.546136] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3635.873783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3635.991342] Lustre: lustre-OST0000: new disk, initializing [ 3635.992936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3635.995351] Lustre: 157875:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3636.028654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3637.937201] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3641.584726] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3642.974311] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3643.895093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3643.898725] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3643.929298] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3644.077686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3645.415921] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3645.490714] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3648.993041] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3648.995479] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3648.996490] LustreError: Skipped 1 previous similar message [ 3649.000486] Lustre: Skipped 1 previous similar message [ 3649.001254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3649.006424] Lustre: Skipped 3 previous similar messages [ 3652.241780] Lustre: server umount lustre-OST0000 complete [ 3654.114250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3654.114885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3654.119852] Lustre: Skipped 2 previous similar messages [ 3654.124608] Lustre: Skipped 1 previous similar message [ 3659.232418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3659.234839] Lustre: Skipped 1 previous similar message [ 3659.685040] Lustre: server umount lustre-MDT0000 complete [ 3660.980385] LustreError: 155901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786483708 with bad export cookie 15966460135711629443 [ 3660.984646] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3661.114809] Lustre: server umount lustre-MDT0001 complete [ 3662.318830] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 17:28:28 (1786483708) [ 3662.868844] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3663.487400] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 17:28:30 (1786483710) [ 3666.017324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3666.186758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3667.506765] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3670.284196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3671.722498] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3674.120736] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3675.939643] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3678.184154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3680.172855] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3682.435939] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3694.611631] LustreError: 162892:0:(ldlm_resource.c:1207:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200000bd0:0x1:0x0].0x61405234 (00000000f4725b33) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3696.097467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3696.099534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3696.101833] Lustre: Skipped 2 previous similar messages [ 3700.770708] Lustre: server umount lustre-MDT0000 complete [ 3701.216787] LustreError: 161019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3701.221523] LustreError: 161019:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3701.998094] LustreError: 161026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786483749 with bad export cookie 15966460135711630311 [ 3702.000371] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3702.003312] LustreError: 161026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3703.096176] LustreError: 160317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3706.335845] LustreError: 160287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3706.336267] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3708.220912] Lustre: server umount lustre-MDT0001 complete [ 3710.466703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3714.822820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3714.907117] LustreError: 164153:0:(llog.c:1655:llog_backup()) MGC192.168.201.138@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 3714.911248] Lustre: 164153:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.138@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 3723.576092] LustreError: 164158:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3728.695822] LustreError: 164159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3728.700798] LustreError: 164159:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3737.056044] LustreError: 164158:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3737.061339] LustreError: 164158:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3754.296213] LustreError: 164159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3754.301746] LustreError: 164159:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 3775.583235] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3775.585031] Lustre: 164153:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.138@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3787.232066] LustreError: 164158:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3787.238419] LustreError: 164158:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 21 previous similar messages [ 3816.423748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3816.425899] Lustre: Skipped 2 previous similar messages [ 3817.732510] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3820.586322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3821.543651] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3821.545940] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xdd9442e64586fd0f [ 3823.051796] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3825.368389] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3825.912609] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3826.663984] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3826.666048] Lustre: Skipped 1 previous similar message [ 3827.681660] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3831.310157] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3836.898389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3836.899425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3836.900326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3836.900331] Lustre: Skipped 5 previous similar messages [ 3836.902436] Lustre: Skipped 2 previous similar messages [ 3837.920174] Lustre: server umount lustre-MDT0000 complete [ 3839.110187] LustreError: 160273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786483886 with bad export cookie 15966460135711636751 [ 3839.113469] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3839.113986] LustreError: 160273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3839.223601] Lustre: server umount lustre-MDT0001 complete [ 3850.710294] Lustre: server umount lustre-OST0000 complete [ 3857.612217] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 3860.743447] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 3873.089969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3873.173872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3873.202516] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3873.240177] Lustre: lustre-MDT0000: new disk, initializing [ 3873.265617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3874.644563] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3878.774597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3878.819232] Lustre: 169225:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3878.925811] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3880.378366] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3884.117411] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3885.900353] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3888.132205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3888.231686] Lustre: 171200:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3888.259445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3888.261516] Lustre: Skipped 3 previous similar messages [ 3890.137745] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3893.833781] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3895.211334] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3896.823443] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3896.854072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3897.311480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3898.685277] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3898.757369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3901.924560] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3901.925156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3901.931705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3901.934297] Lustre: Skipped 1 previous similar message [ 3901.939187] Lustre: Skipped 3 previous similar messages [ 3905.718107] Lustre: server umount lustre-OST0000 complete [ 3909.600801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3909.604074] LustreError: Skipped 1 previous similar message [ 3913.723645] Lustre: server umount lustre-MDT0000 complete [ 3915.513272] LustreError: 169218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786483963 with bad export cookie 15966460135711653544 [ 3915.514976] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3915.522296] LustreError: 169218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3915.671941] Lustre: server umount lustre-MDT0001 complete [ 3919.813931] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 17:32:46 (1786483966) [ 3923.055809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3923.288917] LustreError: 173623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3923.296767] LustreError: 173623:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 23 previous similar messages [ 3925.448246] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3929.479623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3931.751394] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3934.788852] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3938.104374] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3941.621225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3944.861044] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3948.276809] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3961.893266] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 17:33:28 (1786484008) [ 3964.383803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3969.047157] Lustre: server umount lustre-OST0000 complete [ 3975.649784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3975.660074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3975.739966] Lustre: Skipped 5 previous similar messages [ 3975.744918] Lustre: Skipped 6 previous similar messages [ 3976.986575] Lustre: server umount lustre-MDT0000 complete [ 3980.353131] LustreError: 176620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786484028 with bad export cookie 15966460135711654398 [ 3980.360589] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3980.373612] LustreError: 176620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3980.747780] Lustre: server umount lustre-MDT0001 complete [ 3988.737852] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 3991.195145] Key type lgssc unregistered [ 3991.380650] LNet: 177375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3991.385306] LNetError: 177375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3992.423349] LNet: Removed LNI 192.168.201.138@tcp [ 3993.059175] Key type .llcrypt unregistered [ 3993.061296] Key type ._llcrypt unregistered [ 4016.310472] Key type ._llcrypt registered [ 4016.315970] Key type .llcrypt registered [ 4016.637945] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4018.742310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4019.048711] alg: No test for adler32 (adler32-zlib) [ 4020.374574] Lustre: Lustre: Build Version: 2.17.56_52_g54963d0 [ 4020.716732] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 4022.487250] Key type lgssc registered [ 4024.462257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4034.941097] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4034.972929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4050.655366] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4055.776729] LustreError: 178621:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.138@tcp: failed processing log, type 4: rc = -110 [ 4060.960726] LustreError: 178621:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4060.974095] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4060.978930] LustreError: MGC192.168.201.138@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4060.997290] LustreError: 178621:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4061.008853] LustreError: 178621:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 4061.013906] LustreError: 178621:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4061.059345] Lustre: server umount lustre-OST0000 complete [ 4061.061308] LustreError: 178621:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4070.428609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4072.390512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4078.392867] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4082.657704] LustreError: 179121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4082.666916] LustreError: 179121:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4083.690512] LustreError: 179122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4088.806343] LustreError: 179121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4092.132768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4093.138446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4100.352712] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4113.717438] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4121.398903] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4128.054030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4128.360613] LustreError: 180940:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4128.375701] Lustre: 180940:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.138@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4128.571579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4134.805890] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4142.265361] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4149.218706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4149.221989] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4149.240290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4149.248687] Lustre: Skipped 1 previous similar message [ 4149.255076] Lustre: Skipped 2 previous similar messages [ 4150.954108] Lustre: server umount lustre-MDT0000 complete [ 4154.338815] LustreError: 179121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4154.356428] LustreError: 179121:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4154.890574] LustreError: 179102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786484202 with bad export cookie 15921609734439995048 [ 4154.894156] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4154.901338] LustreError: 179102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4155.165198] Lustre: server umount lustre-MDT0001 complete [ 4170.407115] Lustre: server umount lustre-OST0000 complete [ 4182.343772] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 4192.447782] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4231.294254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4231.596662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4231.620755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4231.701644] Lustre: lustre-MDT0000: new disk, initializing [ 4231.831532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4231.852624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4238.062106] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4249.997908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4250.120084] Lustre: 184464:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4250.290324] LustreError: 184476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4250.314908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4250.326040] Lustre: Skipped 1 previous similar message [ 4250.442283] Lustre: lustre-MDT0001: new disk, initializing [ 4250.625621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4250.707140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4250.745549] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4256.533814] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4268.354119] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4276.484896] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4285.699065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4286.132421] Lustre: lustre-OST0000: new disk, initializing [ 4286.139590] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4286.145514] Lustre: 186436:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4286.299533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4296.165565] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4296.448580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4296.471366] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4296.891584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4307.674943] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4312.324033] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4312.505363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4317.595562] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4317.801406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4322.272137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4322.281695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4322.286495] LustreError: Skipped 1 previous similar message [ 4322.299388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4322.314112] Lustre: Skipped 2 previous similar messages [ 4327.402788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4327.413230] Lustre: Skipped 2 previous similar messages [ 4332.513478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4334.047137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4334.232340] Lustre: server umount lustre-OST0000 complete [ 4337.632333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4337.640625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4337.643548] Lustre: Skipped 1 previous similar message [ 4337.644292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4343.470675] Lustre: server umount lustre-MDT0000 complete [ 4346.968693] LustreError: 184457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786484394 with bad export cookie 15921609734439995769 [ 4346.975811] LustreError: 184457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4346.978853] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4347.333537] Lustre: server umount lustre-MDT0001 complete [ 4354.542995] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 17:40:00 (1786484400) [ 4368.114420] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 4370.502236] Key type lgssc unregistered [ 4370.718273] LNet: 189387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4370.722778] LNetError: 189387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4370.734328] LNet: Removed LNI 192.168.201.138@tcp [ 4371.368717] Key type .llcrypt unregistered [ 4371.370598] Key type ._llcrypt unregistered [ 4386.587570] Key type ._llcrypt registered [ 4386.590128] Key type .llcrypt registered [ 4386.676545] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4387.772464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4387.826102] alg: No test for adler32 (adler32-zlib) [ 4388.900912] Lustre: Lustre: Build Version: 2.17.56_52_g54963d0 [ 4389.144163] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 4390.767196] Key type lgssc registered [ 4391.992314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4398.620218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4398.649321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4400.020182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4403.580428] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4409.320329] LustreError: 190605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4409.340109] LustreError: 190605:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4410.602071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4410.815414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4413.965688] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4420.009728] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4425.177486] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4431.862156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4431.959118] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4431.963668] LustreError: 192468:0:(llog.c:1696:llog_backup()) MGC192.168.201.138@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4431.967706] Lustre: 192468:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.138@tcp: can't backup local config lustre-OST0000: rc = -28 [ 4432.076709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4437.943180] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4444.894595] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4449.769445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.774607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4449.787915] Lustre: Skipped 2 previous similar messages [ 4449.804024] Lustre: Skipped 2 previous similar messages [ 4453.117410] Lustre: server umount lustre-MDT0000 complete [ 4454.882737] LustreError: 190605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4454.896180] LustreError: 190605:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4457.029843] LustreError: 190585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786484504 with bad export cookie 10036579882645273547 [ 4457.035895] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4457.045030] LustreError: 190585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4457.416580] Lustre: server umount lustre-MDT0001 complete [ 4470.874037] Lustre: server umount lustre-OST0000 complete [ 4479.934262] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 4488.197269] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4517.023106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4517.247352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4517.264916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4517.334629] Lustre: lustre-MDT0000: new disk, initializing [ 4517.416416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4517.429093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4521.410706] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4530.075628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4530.183760] Lustre: 195989:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4530.225561] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4530.227072] Lustre: Skipped 1 previous similar message [ 4530.231533] LustreError: 196771:0:(llog.c:1696:llog_backup()) MGC192.168.201.138@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 4530.238093] LustreError: 196771:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4530.243217] Lustre: 196771:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.138@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 4530.274960] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4530.278677] Lustre: Skipped 1 previous similar message [ 4530.332695] Lustre: lustre-MDT0001: new disk, initializing [ 4530.396050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4530.446606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4530.459695] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4533.913264] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4540.858241] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4544.650601] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4549.026995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4549.172503] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4549.174797] Lustre: Skipped 1 previous similar message [ 4549.176860] LustreError: 197932:0:(llog.c:1696:llog_backup()) MGC192.168.201.138@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4549.181237] LustreError: 197932:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4549.183957] Lustre: 197932:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.138@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 4549.189349] Lustre: 197932:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 4549.201901] Lustre: lustre-OST0000: new disk, initializing [ 4549.204881] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4549.208867] Lustre: 197956:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4549.266604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4552.994376] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4554.780127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4554.787507] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4554.833956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4559.361984] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4562.066124] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4562.205832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4564.555735] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4564.656804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4570.080112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4570.082758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4570.088423] LustreError: Skipped 1 previous similar message [ 4570.089187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4570.098172] Lustre: Skipped 1 previous similar message [ 4571.797854] Lustre: server umount lustre-OST0000 complete [ 4575.200692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4575.205740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4575.209445] Lustre: Skipped 1 previous similar message [ 4575.211898] Lustre: Skipped 1 previous similar message [ 4580.322694] LustreError: 195998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4580.336095] LustreError: 195998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4581.768136] Lustre: server umount lustre-MDT0000 complete [ 4583.703134] LustreError: 197955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786484631 with bad export cookie 10036579882645274268 [ 4583.707187] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4583.707706] LustreError: 197955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4583.849255] Lustre: server umount lustre-MDT0001 complete [ 4587.767934] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 17:43:54 (1786484634) [ 4594.544145] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4598.330112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4598.468299] LustreError: 201077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4598.486397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4599.999865] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4603.010147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4604.630079] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4605.675067] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4608.184783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4608.226233] LustreError: 202562:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4610.306103] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4613.180981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4613.236504] Lustre: lustre-OST0001: new disk, initializing [ 4613.238235] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4613.240519] Lustre: 203384:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4615.151130] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4615.996478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4615.999899] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4616.027184] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4620.438193] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4622.899851] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4623.490898] Lustre: DEBUG MARKER: ADD OST3 [ 4625.038579] loop: module loaded [ 4625.056526] loop0: detected capacity change from 0 to 204800000 [ 4625.146612] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4625.152198] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4625.156478] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4627.124573] loop0: detected capacity change from 0 to 204800000 [ 4628.908700] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4628.970166] Lustre: lustre-OST0003: new disk, initializing [ 4628.971979] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 4628.974556] Lustre: 205116:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4630.113812] Lustre: DEBUG MARKER: STOP OST3 [ 4630.540508] Lustre: *** cfs_fail_loc=2109, val=2*** [ 4630.583821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 4630.587808] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 4630.597577] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 4630.792839] Lustre: Failing over lustre-OST0003 [ 4630.834517] Lustre: server umount lustre-OST0003 complete [ 4630.918376] LustreError: 202563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4631.009803] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4632.543045] LustreError: 205482:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 4634.994702] Lustre: DEBUG MARKER: START OST3 again [ 4635.617588] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4635.619560] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4635.622314] Lustre: Skipped 1 previous similar message [ 4637.008840] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4637.075409] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 4637.078471] Lustre: Skipped 4 previous similar messages [ 4637.082459] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 4638.625671] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4638.632907] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4638.633061] Lustre: lustre-OST0003: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4638.789411] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4652.157858] Lustre: Failing over lustre-OST0003 [ 4652.204778] Lustre: server umount lustre-OST0003 complete [ 4652.518713] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4652.519424] LustreError: 202564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4652.525584] Lustre: Skipped 1 previous similar message [ 4652.529800] LustreError: 202564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4664.799826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4664.802241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4664.806381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4670.809865] Lustre: server umount lustre-MDT0000 complete [ 4671.967317] LustreError: 201811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786484719 with bad export cookie 10036579882645275150 [ 4671.970567] LustreError: 201811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4671.971431] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4672.994302] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4672.996329] Lustre: Skipped 5 previous similar messages [ 4678.318527] Lustre: server umount lustre-MDT0001 complete [ 4679.641745] Lustre: server umount lustre-OST0000 complete [ 4680.889037] Lustre: server umount lustre-OST0001 complete [ 4682.970307] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 4685.541829] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4697.781340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4697.859973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4697.869877] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4697.905517] Lustre: lustre-MDT0000: new disk, initializing [ 4697.929869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4699.280777] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4703.389608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4703.433730] Lustre: 210525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4703.519684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4703.521774] Lustre: Skipped 1 previous similar message [ 4703.534323] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4704.871678] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4708.593211] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4710.455129] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4712.608981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4712.717785] Lustre: 212494:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4713.975874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4713.978331] Lustre: Skipped 1 previous similar message [ 4713.980418] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4714.016773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4714.599353] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4718.127351] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4719.621216] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4719.686480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4721.157164] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4721.223428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4724.193423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4724.198878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4724.202668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4724.207412] Lustre: Skipped 6 previous similar messages [ 4724.211096] Lustre: Skipped 2 previous similar messages [ 4728.029753] Lustre: server umount lustre-OST0000 complete [ 4734.433702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4734.435757] LustreError: Skipped 1 previous similar message [ 4735.510701] Lustre: server umount lustre-MDT0000 complete [ 4736.769336] LustreError: 210517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786484784 with bad export cookie 10036579882645277397 [ 4736.773565] LustreError: 210517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4736.773906] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4736.918720] Lustre: server umount lustre-MDT0001 complete [ 4738.358678] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 17:46:24 (1786484784) [ 4745.350359] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 4748.865493] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4761.710767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4761.798938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4761.808081] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4761.809746] Lustre: Skipped 3 previous similar messages [ 4761.843908] Lustre: lustre-MDT0000: new disk, initializing [ 4761.845592] Lustre: Skipped 2 previous similar messages [ 4761.869127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4763.133020] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4767.261153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4767.302704] Lustre: 217095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4767.404435] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4768.746646] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4772.564451] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4774.504634] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4776.701435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4776.812181] Lustre: 219063:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4778.467433] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4778.747587] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4782.899217] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4784.489838] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4784.555783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4785.885697] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4785.968830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4788.705882] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4788.706109] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4788.709499] Lustre: Skipped 2 previous similar messages [ 4788.711944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4788.715242] Lustre: Skipped 4 previous similar messages [ 4792.740186] Lustre: server umount lustre-OST0000 complete [ 4800.280241] Lustre: server umount lustre-MDT0000 complete [ 4801.473125] LustreError: 217087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786484849 with bad export cookie 10036579882645278279 [ 4801.476384] LustreError: 217087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4801.476961] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4801.598562] Lustre: server umount lustre-MDT0001 complete [ 4804.399186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4804.567469] LustreError: 221277:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4804.574029] LustreError: 221277:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 4805.945433] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4808.621111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4810.059385] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4812.514180] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4814.386994] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4816.933504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4819.012225] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4821.344962] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4839.904591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4839.906956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4839.907163] LustreError: Skipped 3 previous similar messages [ 4839.910104] Lustre: Skipped 2 previous similar messages [ 4844.315234] Lustre: server umount lustre-OST0000 complete [ 4859.871065] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4859.933914] Lustre: server umount lustre-MDT0000 complete [ 4861.244520] LustreError: 221303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786484909 with bad export cookie 10036579882645279161 [ 4861.248306] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4861.251827] LustreError: 221303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4861.383978] Lustre: server umount lustre-MDT0001 complete [ 4956.660906] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 4957.646191] Key type lgssc unregistered [ 4957.763611] LNet: 224963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4957.768084] LNetError: 224963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4957.776408] LNet: Removed LNI 192.168.201.138@tcp [ 4958.140116] Key type .llcrypt unregistered [ 4958.141895] Key type ._llcrypt unregistered [ 4967.005114] Key type ._llcrypt registered [ 4967.006166] Key type .llcrypt registered [ 4967.038599] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 4973.708389] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4974.184678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4974.195157] alg: No test for adler32 (adler32-zlib) [ 4975.087112] Lustre: Lustre: Build Version: 2.17.56_52_g54963d0 [ 4975.191091] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 4976.799239] Key type lgssc registered [ 4977.274192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4991.407795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4991.413204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4992.498886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4992.508098] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4992.554337] Lustre: lustre-MDT0000: new disk, initializing [ 4992.575651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4992.581547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4993.948617] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4998.368636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4998.414860] Lustre: 228287:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4998.443532] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4998.445047] Lustre: Skipped 1 previous similar message [ 4998.483013] Lustre: lustre-MDT0001: new disk, initializing [ 4998.524451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4998.538674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4998.542467] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5000.401698] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5004.635511] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5006.628795] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5009.090951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5009.233231] Lustre: lustre-OST0000: new disk, initializing [ 5009.235443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5009.238804] Lustre: 230256:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5009.288281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5010.174278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5010.178506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5010.217547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5011.497987] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5015.296854] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5016.774247] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5016.848037] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5018.523406] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5018.648847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5020.642591] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5020.643634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5020.645787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5020.651684] Lustre: Skipped 1 previous similar message [ 5025.493194] Lustre: server umount lustre-OST0000 complete [ 5029.344160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5029.347584] LustreError: Skipped 1 previous similar message [ 5029.349595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5029.355623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5029.358473] Lustre: Skipped 1 previous similar message [ 5030.881203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5030.882499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5036.001448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5036.006273] Lustre: Skipped 2 previous similar messages [ 5041.119221] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5041.122251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5041.126326] Lustre: Skipped 1 previous similar message [ 5041.214322] Lustre: server umount lustre-MDT0000 complete [ 5042.987655] LustreError: 228282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485090 with bad export cookie 18080439036516905807 [ 5042.994603] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5043.135197] Lustre: server umount lustre-MDT0001 complete [ 5050.078336] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5053.955350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5054.131213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5055.563532] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5058.528813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5058.666983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5060.045953] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5061.015962] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5063.258262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5065.628167] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5069.569797] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5069.650558] Lustre: lustre-OST0001: new disk, initializing [ 5069.652183] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5069.654869] Lustre: 235496:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5069.694594] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5069.696753] Lustre: Skipped 1 previous similar message [ 5072.064536] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5075.987232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5075.992492] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5076.050326] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5077.905556] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5085.021884] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5089.341195] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 17:52:15 (1786485135) [ 5091.297817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5091.300293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5091.303186] Lustre: Skipped 3 previous similar messages [ 5091.305248] Lustre: Skipped 2 previous similar messages [ 5092.247798] Lustre: server umount lustre-MDT0000 complete [ 5093.523266] LustreError: 233171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485141 with bad export cookie 18080439036516906675 [ 5093.531971] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5093.532215] LustreError: 233171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5093.679160] Lustre: server umount lustre-MDT0001 complete [ 5104.641621] Lustre: server umount lustre-OST0000 complete [ 5115.303485] Lustre: server umount lustre-OST0001 complete [ 5117.702529] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 5121.435573] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5134.863856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5134.982193] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5134.993601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5135.032433] Lustre: lustre-MDT0000: new disk, initializing [ 5135.057661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5135.065406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5136.322869] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5140.919413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5140.974510] Lustre: 239433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5141.073782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5142.401875] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5146.657219] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5148.537749] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5150.958719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5151.093934] Lustre: lustre-OST0000: new disk, initializing [ 5151.097216] Lustre: Skipped 1 previous similar message [ 5151.100640] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5151.104816] Lustre: Skipped 2 previous similar messages [ 5151.108782] Lustre: 241401:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5152.963453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5152.966490] Lustre: Skipped 1 previous similar message [ 5152.968741] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5153.009942] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5153.259972] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5157.382593] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5158.939196] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5158.998717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5160.589047] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5160.665682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5163.488856] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5163.490961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5163.491690] LustreError: Skipped 1 previous similar message [ 5163.495971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5163.499104] Lustre: Skipped 2 previous similar messages [ 5167.725545] Lustre: server umount lustre-OST0000 complete [ 5171.680500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5171.685427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5171.692988] Lustre: Skipped 1 previous similar message [ 5175.584908] Lustre: server umount lustre-MDT0000 complete [ 5176.798170] LustreError: 239427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485224 with bad export cookie 18080439036516908796 [ 5176.800860] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5176.801258] LustreError: 239427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5176.911565] Lustre: server umount lustre-MDT0001 complete [ 5179.740972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5179.919862] LustreError: 243613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5179.926651] LustreError: 243613:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5179.945735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5179.949405] Lustre: Skipped 2 previous similar messages [ 5181.472385] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5184.702442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5186.537895] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5189.168804] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5191.014289] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5193.190565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5195.786684] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5198.747210] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5348.322374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5348.325470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5348.328254] LustreError: Skipped 1 previous similar message [ 5348.336370] Lustre: Skipped 1 previous similar message [ 5348.339990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5348.345427] Lustre: Skipped 4 previous similar messages [ 5349.645121] Lustre: server umount lustre-OST0000 complete [ 5357.167231] Lustre: server umount lustre-MDT0000 complete [ 5358.560536] LustreError: 244354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5358.568096] LustreError: 244354:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5358.615809] LustreError: 243595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485406 with bad export cookie 18080439036516909678 [ 5358.621142] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5358.622112] LustreError: 243595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5358.755448] Lustre: server umount lustre-MDT0001 complete [ 5464.875920] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 5465.871192] Key type lgssc unregistered [ 5466.004295] LNet: 247221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5466.006808] LNetError: 247221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5466.017316] LNet: Removed LNI 192.168.201.138@tcp [ 5466.319090] Key type .llcrypt unregistered [ 5466.320055] Key type ._llcrypt unregistered [ 5474.176742] Key type ._llcrypt registered [ 5474.177871] Key type .llcrypt registered [ 5474.212251] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 5480.360797] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5480.663992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5480.674273] alg: No test for adler32 (adler32-zlib) [ 5481.533253] Lustre: Lustre: Build Version: 2.17.56_52_g54963d0 [ 5481.626125] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 5483.207180] Key type lgssc registered [ 5483.556733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5498.000827] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5501.424259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5501.430531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5502.504802] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5502.515947] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5502.551599] Lustre: lustre-MDT0000: new disk, initializing [ 5502.571374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5502.578734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5503.928584] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5507.900496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5507.935218] Lustre: 251263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5507.956918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5507.958779] Lustre: Skipped 1 previous similar message [ 5507.991547] Lustre: lustre-MDT0001: new disk, initializing [ 5508.026192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5508.038597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5508.041620] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5509.296388] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5511.532311] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5513.728603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5513.824083] Lustre: lustre-OST0000: new disk, initializing [ 5513.825667] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5513.827750] Lustre: 252890:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5513.856465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5515.634672] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5518.328127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5518.331505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5518.383089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5519.665566] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5519.728180] Lustre: lustre-OST0001: new disk, initializing [ 5519.729566] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5519.731736] Lustre: 253759:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5519.765424] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5521.638372] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5527.045232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5527.049395] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5527.094741] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5527.118461] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5534.897458] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5538.784471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5538.787387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5538.793761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5542.369394] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5542.370413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5542.372643] Lustre: Skipped 2 previous similar messages [ 5542.374567] Lustre: Skipped 2 previous similar messages [ 5544.420198] Lustre: server umount lustre-MDT0000 complete [ 5545.616460] LustreError: 252056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485593 with bad export cookie 12917236921547006802 [ 5545.617405] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5545.621382] LustreError: 252056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5545.737350] Lustre: server umount lustre-MDT0001 complete [ 5556.679886] Lustre: server umount lustre-OST0000 complete [ 5568.155316] Lustre: server umount lustre-OST0001 complete [ 5572.563546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5575.545446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5578.186070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5581.223099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5587.117354] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5590.514897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5590.521780] Lustre: lustre-MDT0000: reset Object Index mappings [ 5590.699418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5592.147402] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5594.924180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5594.932808] Lustre: lustre-MDT0001: reset Object Index mappings [ 5596.462066] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5597.453828] Lustre: 259903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5599.726923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5599.883389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5599.886098] Lustre: Skipped 1 previous similar message [ 5600.932405] LustreError: 260256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5600.937546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5600.938980] LustreError: 260256:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5601.901183] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5604.970485] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5606.956229] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5610.287387] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5610.474454] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 5611.608354] Lustre: 261768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5615.588582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5615.590858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5615.592559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5615.593798] Lustre: Skipped 3 previous similar messages [ 5618.982032] Lustre: server umount lustre-MDT0000 complete [ 5620.373492] LustreError: 261770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485668 with bad export cookie 12917236921547013578 [ 5620.377914] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5620.378601] LustreError: 261770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5620.524567] Lustre: server umount lustre-MDT0001 complete [ 5632.195977] Lustre: server umount lustre-OST0000 complete [ 5643.808043] Lustre: server umount lustre-OST0001 complete [ 5650.419386] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 18:01:36 (1786485696) [ 5656.078847] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 5659.767075] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5672.971447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5673.060566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5673.072331] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5673.108400] Lustre: lustre-MDT0000: new disk, initializing [ 5673.149121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5673.152266] Lustre: Skipped 1 previous similar message [ 5673.160557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5674.623779] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5679.251224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5679.302289] Lustre: 266028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5679.420735] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5680.958795] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5684.931848] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5686.941216] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5689.305117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5689.426797] Lustre: lustre-OST0000: new disk, initializing [ 5689.428531] Lustre: Skipped 1 previous similar message [ 5689.430556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5689.433087] Lustre: Skipped 2 previous similar messages [ 5689.435457] Lustre: 267997:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5691.069539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5691.073417] Lustre: Skipped 1 previous similar message [ 5691.075859] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5691.121041] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5691.809913] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5696.049966] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5697.545089] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5697.632812] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5699.157684] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5699.229459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5701.600757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5701.604346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5701.610074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5701.613121] Lustre: Skipped 4 previous similar messages [ 5706.721583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5706.724343] Lustre: Skipped 1 previous similar message [ 5714.399155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5714.454712] Lustre: server umount lustre-OST0000 complete [ 5716.961040] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5716.961719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5716.964592] Lustre: Skipped 2 previous similar messages [ 5716.966545] Lustre: Skipped 2 previous similar messages [ 5721.816096] Lustre: server umount lustre-MDT0000 complete [ 5722.080752] LustreError: 266041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5722.088028] LustreError: 266041:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5723.150273] LustreError: 266022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485770 with bad export cookie 12917236921547015664 [ 5723.153923] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5723.154451] LustreError: 266022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5723.285820] Lustre: server umount lustre-MDT0001 complete [ 5730.421266] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5733.917043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5734.046248] LustreError: 270924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5734.059189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5734.062398] Lustre: Skipped 2 previous similar messages [ 5735.467771] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5738.536795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5740.169888] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5741.232608] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5743.571296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5745.832568] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5748.855148] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5748.928090] Lustre: lustre-OST0001: new disk, initializing [ 5748.930292] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5748.933476] Lustre: 273230:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5750.154803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5750.159245] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5750.186962] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5751.117871] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5755.821186] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5757.133970] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5759.118082] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 5759.132644] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 5764.576115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5764.579202] LustreError: Skipped 1 previous similar message [ 5764.581347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5764.585596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5764.587905] Lustre: Skipped 1 previous similar message [ 5767.258536] Lustre: server umount lustre-MDT0000 complete [ 5768.516154] LustreError: 274399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485816 with bad export cookie 12917236921547016546 [ 5768.518812] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5768.519920] LustreError: 274399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5768.631208] Lustre: server umount lustre-MDT0001 complete [ 5779.912491] Lustre: server umount lustre-OST0000 complete [ 5790.365796] Lustre: server umount lustre-OST0001 complete [ 5796.108511] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5799.875376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5800.011996] LustreError: 276281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5800.032985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5800.035241] Lustre: Skipped 3 previous similar messages [ 5801.558474] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5804.675320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5806.179609] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5807.161160] Lustre: 277421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5809.476714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5810.659622] LustreError: 277774:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5810.665031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 5811.557578] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5814.381392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5815.532894] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 5816.397073] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5819.510094] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5820.693976] Lustre: 279293:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5822.375101] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 5822.387142] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 5825.450362] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 18:04:32 (1786485872) [ 5840.863839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5840.866790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5840.871183] Lustre: Skipped 3 previous similar messages [ 5840.873591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5840.875677] Lustre: Skipped 4 previous similar messages [ 5843.028719] Lustre: server umount lustre-MDT0000 complete [ 5844.194241] LustreError: 276263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485891 with bad export cookie 12917236921547022867 [ 5844.197063] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5844.198985] LustreError: 276263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5846.496019] LustreError: 276278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5846.501439] LustreError: 276278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5850.351687] Lustre: server umount lustre-MDT0001 complete [ 5857.662787] Lustre: server umount lustre-OST0000 complete [ 5873.119116] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5873.182508] Lustre: server umount lustre-OST0001 complete [ 5875.249365] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 5878.446102] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5893.011877] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5896.245569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5896.322820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5896.332642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5896.365831] Lustre: lustre-MDT0000: new disk, initializing [ 5896.387821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5897.708304] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5901.615079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5901.651652] Lustre: 283524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5901.753564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5902.982582] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5907.426027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5907.522816] Lustre: 285155:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5908.323467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5909.525505] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5913.683083] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5915.692756] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5915.739056] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5920.263921] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5925.594741] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 18:06:12 (1786485972) [ 5927.391995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5927.395786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5927.399843] Lustre: Skipped 5 previous similar messages [ 5927.401609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5927.404340] Lustre: Skipped 8 previous similar messages [ 5932.516539] Lustre: server umount lustre-MDT0000 complete [ 5933.806091] LustreError: 283515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485981 with bad export cookie 12917236921547030756 [ 5933.809486] LustreError: 283515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5933.809551] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5933.932613] Lustre: server umount lustre-MDT0001 complete [ 5944.257910] Lustre: server umount lustre-OST0000 complete [ 5955.687748] Lustre: server umount lustre-OST0001 complete [ 5958.429085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5958.576278] LustreError: 288267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5958.583873] LustreError: 288267:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5958.607591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5958.610164] Lustre: Skipped 7 previous similar messages [ 5959.892424] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5962.624266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5964.081296] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5966.394725] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5968.109651] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5970.232561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5972.332768] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5974.674139] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5990.785769] loop1: detected capacity change from 0 to 204800000 [ 5990.858244] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5990.863776] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5990.867072] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5992.507121] loop1: detected capacity change from 0 to 204800000 [ 5993.954099] LustreError: 290096:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5993.958458] LustreError: 290096:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 5994.325798] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5994.355888] LustreError: 288256:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs11968: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 5994.359883] LustreError: 288256:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs11968 [ 5994.361468] LustreError: 288256:0:(mgs_handler.c:517:mgs_target_reg()) fs11968-MDT0001: Error checking target index: rc = -13 [ 5994.364134] LustreError: fs11968-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5994.366617] LustreError: 292007:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 5994.368661] LustreError: 292007:0:(tgt_mount.c:2129:server_put_super()) no obd fs11968-MDT0001 [ 5994.370798] LustreError: 292007:0:(tgt_mount.c:116:server_deregister_mount()) fs11968-MDT0001 not registered [ 5994.380932] Lustre: server umount fs11968-MDT0001 complete [ 5994.382402] LustreError: 292007:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5997.117241] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5997.146269] Lustre: Setting parameter fs11968-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs11968-MDT0001 [ 5997.149956] Lustre: Skipped 2 previous similar messages [ 5997.165149] Lustre: srv-fs11968-MDT0001: No data found on store. Initialize space. [ 5997.167301] Lustre: Skipped 4 previous similar messages [ 5997.203688] Lustre: fs11968-MDT0001: new disk, initializing [ 5997.205166] Lustre: Skipped 3 previous similar messages [ 5997.236890] Lustre: srv-fs11968-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 5997.240335] Lustre: 292471:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs11968-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5997.243925] LustreError: 292471:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs11968-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5998.495581] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6001.284313] Lustre: server umount fs11968-MDT0001 complete [ 6005.371306] loop1: detected capacity change from 0 to 204800000 [ 6007.178969] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6007.300273] Lustre: 294037:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs11968-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6007.303149] LustreError: 294037:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs11968-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6008.659291] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6010.312696] Lustre: server umount fs11968-MDT0001 complete [ 6013.919802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6013.921609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6013.925555] Lustre: Skipped 3 previous similar messages [ 6019.539668] Lustre: server umount lustre-MDT0000 complete [ 6022.191071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6022.232072] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6023.672209] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6025.995150] loop1: detected capacity change from 0 to 204800000 [ 6027.734471] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6027.750528] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6027.850920] Lustre: 296479:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs11968-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6027.853987] LustreError: 296479:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs11968-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6029.050518] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6030.505636] Lustre: server umount fs11968-MDT0001 complete [ 6042.702441] Lustre: server umount lustre-OST0000 complete [ 6048.224805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6048.227380] LustreError: Skipped 1 previous similar message [ 6050.009535] Lustre: server umount lustre-MDT0000 complete [ 6051.205284] LustreError: 294527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786486098 with bad export cookie 12917236921547033430 [ 6051.209237] LustreError: 294527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6051.320798] Lustre: server umount lustre-MDT0001 complete [ 6054.844440] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6055.816229] Key type lgssc unregistered [ 6055.947358] LNet: 298819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6055.949394] LNetError: 298819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6055.958283] LNet: Removed LNI 192.168.201.138@tcp [ 6056.256127] Key type .llcrypt unregistered [ 6056.257638] Key type ._llcrypt unregistered [ 6063.820373] Key type ._llcrypt registered [ 6063.821435] Key type .llcrypt registered [ 6063.864958] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6064.258578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6064.263464] alg: No test for adler32 (adler32-zlib) [ 6065.101162] Lustre: Lustre: Build Version: 2.17.56_52_g54963d0 [ 6065.183657] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 6066.759110] Key type lgssc registered [ 6067.094664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6069.577942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6069.583280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6070.718668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6072.037991] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6074.879727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6075.016678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6076.324238] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6078.743801] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6080.689171] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6083.085126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6083.209723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6084.259857] LustreError: 301910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6084.270930] LustreError: 301910:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6085.158489] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6087.527418] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6089.702508] LustreError: 301910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6094.819853] LustreError: 302092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6096.865779] LustreError: 302092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6096.870668] LustreError: 302092:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6101.986300] LustreError: 301910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6101.991638] LustreError: 301910:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6104.207177] loop1: detected capacity change from 0 to 204800000 [ 6104.269503] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6104.273617] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6104.277974] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6105.941321] loop1: detected capacity change from 0 to 204800000 [ 6107.673974] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6107.721676] Lustre: fs26413-OST0000: new disk, initializing [ 6107.723727] Lustre: srv-fs26413-OST0000: No data found on store. Initialize space. [ 6107.726274] Lustre: 303836:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6107.744401] Lustre: fs26413-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6109.450808] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6112.226762] LustreError: 301911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6112.231680] LustreError: 301911:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6112.699675] Lustre: server umount fs26413-OST0000 complete [ 6116.486159] loop1: detected capacity change from 0 to 204800000 [ 6118.268394] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6118.333122] Lustre: fs26413-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6120.261407] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6123.023207] loop2: detected capacity change from 0 to 204800000 [ 6123.099851] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6123.103715] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6123.107667] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6124.796328] loop2: detected capacity change from 0 to 204800000 [ 6126.540715] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6126.569315] LustreError: fs26413-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 6126.573720] LustreError: fs26413-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6126.577821] LustreError: 306873:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 6126.580191] LustreError: 306873:0:(tgt_mount.c:2129:server_put_super()) no obd fs26413-OST0001 [ 6126.582614] LustreError: 306873:0:(tgt_mount.c:116:server_deregister_mount()) fs26413-OST0001 not registered [ 6126.592286] Lustre: server umount fs26413-OST0001 complete [ 6126.593735] LustreError: 306873:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6128.952387] LustreError: 302092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6128.956703] LustreError: 302092:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6129.861522] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6129.900471] Lustre: fs26413-OST0001: new disk, initializing [ 6129.902273] Lustre: srv-fs26413-OST0001: No data found on store. Initialize space. [ 6129.904640] Lustre: 307424:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6129.927093] Lustre: fs26413-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6131.752232] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6135.119770] Lustre: server umount fs26413-OST0000 complete [ 6138.088791] Lustre: server umount fs26413-OST0001 complete [ 6141.849864] loop1: detected capacity change from 0 to 204800000 [ 6143.574119] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6145.356151] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6147.387412] loop2: detected capacity change from 0 to 204800000 [ 6149.079709] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6149.137509] Lustre: fs26413-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6149.139336] Lustre: Skipped 1 previous similar message [ 6150.892130] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6153.565527] Lustre: server umount fs26413-OST0000 complete [ 6156.445452] Lustre: server umount fs26413-OST0001 complete [ 6161.375850] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6161.377535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6161.380919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6163.424687] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6163.426259] LustreError: 301911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6163.426467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6163.426760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6163.431649] LustreError: 301911:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 6168.545123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6168.546867] Lustre: Skipped 2 previous similar messages [ 6173.665623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6173.667169] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 6173.667499] Lustre: Skipped 1 previous similar message [ 6174.175145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6174.222983] Lustre: server umount lustre-OST0000 complete [ 6177.759773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6177.763103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6177.767875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6181.529191] Lustre: server umount lustre-MDT0000 complete [ 6182.700621] LustreError: 300064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786486230 with bad export cookie 1736941765707493606 [ 6182.703864] LustreError: 300064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6182.704398] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6182.817497] Lustre: server umount lustre-MDT0001 complete [ 6186.389145] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6187.371708] Key type lgssc unregistered [ 6187.489363] LNet: 313338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6187.492236] LNetError: 313338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6187.502243] LNet: Removed LNI 192.168.201.138@tcp [ 6187.824140] Key type .llcrypt unregistered [ 6187.824992] Key type ._llcrypt unregistered [ 6197.405941] Key type ._llcrypt registered [ 6197.407365] Key type .llcrypt registered [ 6197.442396] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 6203.487074] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6203.881171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6203.893959] alg: No test for adler32 (adler32-zlib) [ 6204.736321] Lustre: Lustre: Build Version: 2.17.56_52_g54963d0 [ 6204.817155] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 6206.399124] Key type lgssc registered [ 6206.739822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6217.553108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6217.558477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6218.629876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6218.647921] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6218.680764] Lustre: lustre-MDT0000: new disk, initializing [ 6218.700298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6218.705417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6220.012657] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6223.913060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6223.950096] Lustre: 317166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6223.970441] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6223.971958] Lustre: Skipped 1 previous similar message [ 6224.001158] Lustre: lustre-MDT0001: new disk, initializing [ 6224.031362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6224.043088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6224.046430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6225.263066] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6228.780792] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6230.630700] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6232.939339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6233.045094] Lustre: lustre-OST0000: new disk, initializing [ 6233.047460] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6233.050764] Lustre: 319135:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6233.083500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6235.011525] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6238.200841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6238.206585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6238.236164] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6238.755063] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6239.438652] Lustre: Mounted lustre-client - version 2.17.56_52_g54963d0 [ 6239.769173] Lustre: Unmounted lustre-client [ 6243.296746] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6243.297059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6243.300137] LustreError: Skipped 1 previous similar message [ 6243.305746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6243.308352] Lustre: Skipped 1 previous similar message [ 6244.755119] Lustre: server umount lustre-OST0000 complete [ 6249.952759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6249.953197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6249.961698] Lustre: Skipped 1 previous similar message [ 6249.964103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6249.965784] Lustre: Skipped 1 previous similar message [ 6252.125440] Lustre: server umount lustre-MDT0000 complete [ 6253.358598] LustreError: 317159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786486301 with bad export cookie 13379453786782534826 [ 6253.361551] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6253.361866] LustreError: 317159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6253.479029] Lustre: server umount lustre-MDT0001 complete [ 6257.108871] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6258.035235] Key type lgssc unregistered [ 6296.991200] LNet: 1 local NIs in recovery (showing 1): 192.168.201.138@tcp [ 6297.055329] LNet: 321297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6297.058221] LNetError: 321297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6370.789492] LNet: Removed LNI 192.168.201.138@tcp [ 6371.168139] Key type .llcrypt unregistered [ 6371.169209] Key type ._llcrypt unregistered [ 6377.255752] Key type ._llcrypt registered [ 6377.256913] Key type .llcrypt registered [ 6377.297874] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 6383.749537] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6384.285639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6384.304369] alg: No test for adler32 (adler32-zlib) [ 6385.180267] Lustre: Lustre: Build Version: 2.17.56_52_g54963d0 [ 6385.285577] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 6386.879169] Key type lgssc registered [ 6387.284085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6396.945954] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 18:14:03 (1786486443) [ 6399.107201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6399.113022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6400.195236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6400.207095] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6400.242216] Lustre: lustre-MDT0000: new disk, initializing [ 6400.261857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6400.267159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6401.556034] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6405.672322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6405.713805] Lustre: 324498:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6405.739597] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6405.741512] Lustre: Skipped 1 previous similar message [ 6405.779967] Lustre: lustre-MDT0001: new disk, initializing [ 6405.814123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6405.828968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6405.832583] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6407.178968] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6410.819901] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6412.633497] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6414.881624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6414.987123] Lustre: lustre-OST0000: new disk, initializing [ 6414.988785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6414.991214] Lustre: 326466:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6415.023249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6416.990353] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6420.698383] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6422.018771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6422.023408] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6422.063810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6432.228762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6432.229248] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6432.230188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6432.234670] Lustre: Skipped 1 previous similar message [ 6432.237125] LustreError: Skipped 1 previous similar message [ 6437.347415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6437.349485] Lustre: Skipped 2 previous similar messages [ 6437.597400] Lustre: server umount lustre-OST0000 complete [ 6441.951984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6441.955744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6441.960591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6447.584502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6447.586491] Lustre: Skipped 3 previous similar messages [ 6452.704859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6452.707175] Lustre: Skipped 1 previous similar message [ 6453.215148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6453.287508] Lustre: server umount lustre-MDT0000 complete [ 6454.580455] LustreError: 324490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786486502 with bad export cookie 17591967064293558119 [ 6454.584578] LustreError: 324490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6454.585464] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6454.712680] Lustre: server umount lustre-MDT0001 complete [ 6458.519359] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6459.519512] Key type lgssc unregistered [ 6459.652385] LNet: 328377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6459.655223] LNetError: 328377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6459.666333] LNet: Removed LNI 192.168.201.138@tcp [ 6459.983911] Key type .llcrypt unregistered [ 6459.985095] Key type ._llcrypt unregistered [ 6468.251723] Key type ._llcrypt registered [ 6468.252797] Key type .llcrypt registered [ 6468.295275] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6468.729193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6468.801096] alg: No test for adler32 (adler32-zlib) [ 6469.658588] Lustre: Lustre: Build Version: 2.17.56_52_g54963d0 [ 6469.761838] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 6471.343127] Key type lgssc registered [ 6471.701382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6474.415806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6474.422059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6475.589501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6476.960459] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6479.764141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6479.877570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6481.227901] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6483.726117] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6485.632389] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6487.905572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6488.039412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6490.025339] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6492.388115] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6506.052341] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6506.622290] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6510.562938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6510.563054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6510.563940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6510.566751] Lustre: Skipped 2 previous similar messages [ 6512.958988] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.38@tcp (stopping) [ 6512.961536] Lustre: Skipped 2 previous similar messages [ 6515.680908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6515.683018] Lustre: Skipped 2 previous similar messages [ 6518.071920] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.38@tcp (stopping) [ 6522.335109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6522.401111] Lustre: server umount lustre-MDT0000 complete [ 6523.191955] LustreError: 329627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6523.197043] LustreError: 329627:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6523.659210] LustreError: 330365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786486571 with bad export cookie 13077244205300385090 [ 6523.661468] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6523.664561] LustreError: 330365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6525.920434] LustreError: 329626:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6525.921632] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6525.934470] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6525.936739] Lustre: Skipped 3 previous similar messages [ 6528.312097] LustreError: 329628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6529.851884] Lustre: server umount lustre-MDT0001 complete [ 6531.185105] Lustre: server umount lustre-OST0000 complete [ 6534.203838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6535.585282] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6536.550666] Lustre: 333827:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.138@tcp [ 6537.244272] Lustre: server umount MGS complete [ 6540.063160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6540.218453] LustreError: 334398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6540.223909] LustreError: 334398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6540.242246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6541.611708] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6544.515667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6544.652981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6546.118084] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6548.987939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6551.049031] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6553.517226] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6554.069193] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 6555.836834] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6559.452051] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 6560.737501] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6560.737631] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6560.740364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6560.744237] Lustre: Skipped 1 previous similar message [ 6560.747025] Lustre: Skipped 3 previous similar messages [ 6566.755468] Lustre: server umount lustre-OST0000 complete [ 6570.464255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6570.468254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6574.164477] Lustre: server umount lustre-MDT0000 complete [ 6575.388386] LustreError: 334379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786486623 with bad export cookie 13077244205300385916 [ 6575.393906] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6575.394662] LustreError: 334379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6575.527230] Lustre: server umount lustre-MDT0001 complete [ 6579.345559] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6580.374486] Key type lgssc unregistered [ 6580.515419] LNet: 337833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6580.517918] LNetError: 337833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6580.527323] LNet: Removed LNI 192.168.201.138@tcp [ 6580.858851] Key type .llcrypt unregistered [ 6580.859901] Key type ._llcrypt unregistered [ 6587.274287] Key type ._llcrypt registered [ 6587.275285] Key type .llcrypt registered [ 6587.324250] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 6593.874906] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6594.416174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6594.426991] alg: No test for adler32 (adler32-zlib) [ 6595.289937] Lustre: Lustre: Build Version: 2.17.56_52_g54963d0 [ 6595.387063] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 6596.967184] Key type lgssc registered [ 6597.346629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6608.995887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6609.001734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6610.098668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6610.109952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6610.148261] Lustre: lustre-MDT0000: new disk, initializing [ 6610.170155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6610.176540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6611.623421] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6615.806874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6615.847239] Lustre: 340967:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6615.871561] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6615.874249] Lustre: Skipped 1 previous similar message [ 6615.915050] Lustre: lustre-MDT0001: new disk, initializing [ 6615.948627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6615.960166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6615.963181] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6617.345026] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6621.140486] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6623.092091] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6625.366296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6625.468111] Lustre: lustre-OST0000: new disk, initializing [ 6625.469648] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6625.471863] Lustre: 342933:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6625.498300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6626.175808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6626.181502] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6626.218247] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6627.428661] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6631.211143] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6632.679823] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6632.751867] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6634.114288] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6634.188230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6636.514624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6636.515098] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6636.516852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6636.517175] LustreError: Skipped 1 previous similar message [ 6636.520854] Lustre: Skipped 1 previous similar message [ 6641.634951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6641.637793] Lustre: Skipped 2 previous similar messages [ 6646.752966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6649.311121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6649.363813] Lustre: server umount lustre-OST0000 complete [ 6651.872996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6651.873517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6651.876235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6651.880110] Lustre: Skipped 1 previous similar message [ 6651.883965] Lustre: Skipped 2 previous similar messages [ 6656.728155] Lustre: server umount lustre-MDT0000 complete [ 6657.953161] LustreError: 341760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786486705 with bad export cookie 16037671707929687736 [ 6657.956226] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6657.956676] LustreError: 341760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6658.069870] Lustre: server umount lustre-MDT0001 complete [ 6661.023335] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 18:18:27 (1786486707) [ 6666.585749] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 6669.968641] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6681.740036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6681.818555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6681.827828] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6681.862254] Lustre: lustre-MDT0000: new disk, initializing [ 6681.881683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6681.886887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6683.267142] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6687.394712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6687.437187] Lustre: 347686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6687.459766] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6687.462537] Lustre: Skipped 1 previous similar message [ 6687.505160] Lustre: lustre-MDT0001: new disk, initializing [ 6687.539425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6687.551129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6687.554250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6688.875426] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6692.625532] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6694.440798] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6696.726202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6696.825377] Lustre: lustre-OST0000: new disk, initializing [ 6696.827101] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6696.830245] Lustre: 349654:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6696.858573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6697.110448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6697.113567] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6697.153957] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6698.736812] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6702.484537] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6712.800717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6712.802447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6712.803299] LustreError: Skipped 1 previous similar message [ 6712.807498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6712.809420] Lustre: Skipped 1 previous similar message [ 6718.036757] Lustre: server umount lustre-OST0000 complete [ 6723.041292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6723.042242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6723.046441] Lustre: Skipped 2 previous similar messages [ 6723.049064] Lustre: Skipped 2 previous similar messages [ 6725.420580] Lustre: server umount lustre-MDT0000 complete [ 6726.836906] LustreError: 347678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786486774 with bad export cookie 16037671707929688604 [ 6726.842184] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6726.842779] LustreError: 347678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6726.965205] Lustre: server umount lustre-MDT0001 complete [ 6730.666898] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6731.781279] Key type lgssc unregistered [ 6731.910563] LNet: 351805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6731.913826] LNetError: 351805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6731.923496] LNet: Removed LNI 192.168.201.138@tcp [ 6732.257931] Key type .llcrypt unregistered [ 6732.259264] Key type ._llcrypt unregistered [ 6766.183631] Key type ._llcrypt registered [ 6766.184847] Key type .llcrypt registered [ 6766.234139] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6766.372208] Key type .llcrypt unregistered [ 6766.373377] Key type ._llcrypt unregistered [ 6795.069920] Key type ._llcrypt registered [ 6795.070968] Key type .llcrypt registered [ 6795.115176] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6795.240352] Key type .llcrypt unregistered [ 6795.241445] Key type ._llcrypt unregistered [ 6806.707354] Key type ._llcrypt registered [ 6806.708338] Key type .llcrypt registered [ 6806.755744] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6806.917319] Key type .llcrypt unregistered [ 6806.919575] Key type ._llcrypt unregistered [ 6819.220770] Key type ._llcrypt registered [ 6819.221832] Key type .llcrypt registered [ 6819.266277] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6819.396232] Key type .llcrypt unregistered [ 6819.397255] Key type ._llcrypt unregistered [ 6847.733193] Key type ._llcrypt registered [ 6847.734671] Key type .llcrypt registered [ 6847.774892] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6847.916232] Key type .llcrypt unregistered [ 6847.917782] Key type ._llcrypt unregistered [ 6858.951379] Key type ._llcrypt registered [ 6858.952407] Key type .llcrypt registered [ 6858.995446] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6859.123222] Key type .llcrypt unregistered [ 6859.124204] Key type ._llcrypt unregistered [ 6870.440520] Key type ._llcrypt registered [ 6870.441526] Key type .llcrypt registered [ 6870.481403] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6870.598334] Key type .llcrypt unregistered [ 6870.599431] Key type ._llcrypt unregistered [ 6881.469190] Key type ._llcrypt registered [ 6881.470167] Key type .llcrypt registered [ 6881.513208] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6881.621306] Key type .llcrypt unregistered [ 6881.622447] Key type ._llcrypt unregistered [ 6893.118749] Key type ._llcrypt registered [ 6893.119781] Key type .llcrypt registered [ 6893.163675] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6893.283299] Key type .llcrypt unregistered [ 6893.284812] Key type ._llcrypt unregistered [ 6904.245213] Key type ._llcrypt registered [ 6904.246140] Key type .llcrypt registered [ 6904.285746] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6904.401456] Key type .llcrypt unregistered [ 6904.402545] Key type ._llcrypt unregistered [ 6925.597817] Key type ._llcrypt registered [ 6925.598698] Key type .llcrypt registered [ 6925.637202] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6925.754401] Key type .llcrypt unregistered [ 6925.755435] Key type ._llcrypt unregistered [ 6936.900555] Key type ._llcrypt registered [ 6936.901581] Key type .llcrypt registered [ 6936.936212] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6937.045242] Key type .llcrypt unregistered [ 6937.046778] Key type ._llcrypt unregistered [ 6948.076178] Key type ._llcrypt registered [ 6948.078016] Key type .llcrypt registered [ 6948.119778] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6948.219333] Key type .llcrypt unregistered [ 6948.220399] Key type ._llcrypt unregistered [ 6967.280654] Key type ._llcrypt registered [ 6967.281839] Key type .llcrypt registered [ 6967.333182] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 6973.950920] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6974.441987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6974.507484] alg: No test for adler32 (adler32-zlib) [ 6975.381977] Lustre: Lustre: Build Version: 2.17.56_52_g54963d0 [ 6975.473948] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 6977.055191] Key type lgssc registered [ 6977.463372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6989.148983] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6989.154183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6990.243441] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 6990.253748] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 6990.289949] Lustre: tst250fs-MDT0000: new disk, initializing [ 6990.311397] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6990.318970] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6991.746480] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6996.047855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6996.100365] Lustre: 371761:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6996.124787] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 6996.126924] Lustre: Skipped 1 previous similar message [ 6996.162889] Lustre: tst250fs-MDT0001: new disk, initializing [ 6996.196421] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6996.208505] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6996.213251] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6997.502639] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6999.778798] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7001.959124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7002.056514] Lustre: tst250fs-OST0000: new disk, initializing [ 7002.058165] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 7002.061176] Lustre: 373385:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7002.095897] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7003.994120] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7005.050552] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7005.053816] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7005.106353] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7008.228309] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7008.305288] Lustre: tst250fs-OST0001: new disk, initializing [ 7008.308217] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 7008.312892] Lustre: 374260:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7008.361360] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7010.413619] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7013.005678] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 7014.774981] Lustre: 371763:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.138@tcp: IR log tst250fs-mdtir failed, not fatal: rc = -2 [ 7024.101342] Lustre: tst250fs-MDT0000-lwp-OST0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7024.103339] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7024.108219] Lustre: Skipped 2 previous similar messages [ 7024.111651] Lustre: Skipped 1 previous similar message [ 7029.216331] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7029.218158] Lustre: Skipped 4 previous similar messages [ 7029.536556] Lustre: server umount tst250fs-MDT0000 complete [ 7030.943149] LustreError: 371751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786487078 with bad export cookie 1062869056107612637 [ 7030.949005] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7031.074133] Lustre: server umount tst250fs-MDT0001 complete [ 7042.562610] Lustre: server umount tst250fs-OST0000 complete [ 7054.187329] Lustre: server umount tst250fs-OST0001 complete [ 7060.823799] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 7063.704504] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 7074.596656] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 18:25:21 (1786487121) [ 7075.193781] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 7081.382406] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 7084.934576] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 7095.657720] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6763 sec ========== 18:25:42 (1786487142) [ 7096.228887] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:25:42 (1786487142) === [ 7097.440655] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:25:44 (1786487144) === [ 7110.110871] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 7111.094667] Key type lgssc unregistered [ 7111.223408] LNet: 383364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7111.225691] LNetError: 383364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7111.232351] LNet: Removed LNI 192.168.201.138@tcp [ 7111.565102] Key type .llcrypt unregistered [ 7111.566135] Key type ._llcrypt unregistered