[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 534159447 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002000] Switched APIC routing to physical x2apic. [ 0.002013] kvm-guest: setup PV IPIs [ 0.005516] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.006021] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.007013] pid_max: default: 32768 minimum: 301 [ 0.008124] LSM: Security Framework initializing [ 0.009000] Yama: becoming mindful. [ 0.009055] SELinux: Initializing. [ 0.010068] *** VALIDATE selinux *** [ 0.018794] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024585] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025214] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026195] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027143] *** VALIDATE tmpfs *** [ 0.029132] *** VALIDATE proc *** [ 0.030248] *** VALIDATE cgroup *** [ 0.031009] *** VALIDATE cgroup2 *** [ 0.033006] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034167] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036030] Spectre V2 : User space: Vulnerable [ 0.037006] Speculative Store Bypass: Vulnerable [ 0.040421] debug: unmapping init [mem 0xffffffff95059000-0xffffffff95060fff] [ 0.043097] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044598] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045023] ... version: 2 [ 0.046009] ... bit width: 48 [ 0.047008] ... generic registers: 4 [ 0.048008] ... value mask: 0000ffffffffffff [ 0.049008] ... max period: 00007fffffffffff [ 0.050008] ... fixed-purpose events: 3 [ 0.051009] ... event mask: 000000070000000f [ 0.052304] rcu: Hierarchical SRCU implementation. [ 0.054383] smp: Bringing up secondary CPUs ... [ 0.055495] x86: Booting SMP configuration: [ 0.056019] .... node #0, CPUs: #1 #2 #3 [ 0.063113] smp: Brought up 1 node, 4 CPUs [ 0.065019] smpboot: Max logical packages: 1 [ 0.066018] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.128521] node 0 deferred pages initialised in 58ms [ 0.133734] devtmpfs: initialized [ 0.134000] x86/mm: Memory block size: 128MB [ 0.140210] gcov: version magic: 0x41383552 [ 0.143000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.144063] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.145314] pinctrl core: initialized pinctrl subsystem [ 0.146231] [ 0.146740] ************************************************************* [ 0.147012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.148009] ** ** [ 0.149010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.150011] ** ** [ 0.151010] ** This means that this kernel is built to expose internal ** [ 0.152009] ** IOMMU data structures, which may compromise security on ** [ 0.153011] ** your system. ** [ 0.154007] ** ** [ 0.155007] ** If you see this message and you are not debugging the ** [ 0.156009] ** kernel, report this immediately to your vendor! ** [ 0.157007] ** ** [ 0.158010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159015] ************************************************************* [ 0.160815] NET: Registered protocol family 16 [ 0.161613] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.162140] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.163058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.165667] cpuidle: using governor menu [ 0.168770] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.172376] PCI: Using configuration type 1 for base access [ 0.175146] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.186346] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.187022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.189627] cryptd: max_cpu_qlen set to 1000 [ 0.194725] ACPI: Added _OSI(Module Device) [ 0.195000] ACPI: Added _OSI(Processor Device) [ 0.195000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.196018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.201170] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.210498] ACPI: Interpreter enabled [ 0.211055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.212012] ACPI: Using IOAPIC for interrupt routing [ 0.213115] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.214494] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.225668] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.227023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.228104] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.230455] acpiphp: Slot [2] registered [ 0.231114] acpiphp: Slot [3] registered [ 0.232119] acpiphp: Slot [4] registered [ 0.233089] acpiphp: Slot [5] registered [ 0.234138] acpiphp: Slot [6] registered [ 0.235135] acpiphp: Slot [7] registered [ 0.236109] acpiphp: Slot [8] registered [ 0.237246] acpiphp: Slot [9] registered [ 0.238169] acpiphp: Slot [10] registered [ 0.239088] acpiphp: Slot [11] registered [ 0.240203] acpiphp: Slot [12] registered [ 0.241077] acpiphp: Slot [13] registered [ 0.242096] acpiphp: Slot [14] registered [ 0.243070] acpiphp: Slot [15] registered [ 0.244078] acpiphp: Slot [16] registered [ 0.245097] acpiphp: Slot [17] registered [ 0.246119] acpiphp: Slot [18] registered [ 0.247122] acpiphp: Slot [19] registered [ 0.248144] acpiphp: Slot [20] registered [ 0.249137] acpiphp: Slot [21] registered [ 0.250126] acpiphp: Slot [22] registered [ 0.251132] acpiphp: Slot [23] registered [ 0.252067] acpiphp: Slot [24] registered [ 0.253122] acpiphp: Slot [25] registered [ 0.255091] acpiphp: Slot [26] registered [ 0.256090] acpiphp: Slot [27] registered [ 0.257244] acpiphp: Slot [28] registered [ 0.258189] acpiphp: Slot [29] registered [ 0.259142] acpiphp: Slot [30] registered [ 0.260137] acpiphp: Slot [31] registered [ 0.261055] PCI host bridge to bus 0000:00 [ 0.262025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.263036] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.264031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.265028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.266024] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.267023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.268209] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.271840] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.275179] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.281700] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.284071] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.285015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.286015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.287015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.288677] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.289907] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.290036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.291857] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.293010] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.300029] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.303011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.305000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.315036] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.331026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.353017] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.366389] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.372017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.377016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.388017] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.397344] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.404015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.410016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.423018] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.432385] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.439013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.443012] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.455024] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.468179] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.472013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.475014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.483011] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.491277] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.496015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.500013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.506015] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.517142] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.519326] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.521318] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.523259] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.524180] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.526138] iommu: Default domain type: Passthrough [ 0.527443] SCSI subsystem initialized [ 0.528157] ACPI: bus type USB registered [ 0.531000] usbcore: registered new interface driver usbfs [ 0.531000] usbcore: registered new interface driver hub [ 0.531000] usbcore: registered new device driver usb [ 0.534157] pps_core: LinuxPPS API ver. 1 registered [ 0.535007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.543055] PTP clock support registered [ 0.556154] EDAC MC: Ver: 3.0.0 [ 0.561153] PCI: Using ACPI for IRQ routing [ 0.562853] NetLabel: Initializing [ 0.567019] NetLabel: domain hash size = 128 [ 0.570010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.574150] NetLabel: unlabeled traffic allowed by default [ 0.579091] vgaarb: loaded [ 0.581159] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.582000] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.589144] clocksource: Switched to clocksource kvm-clock [ 0.733336] VFS: Disk quotas dquot_6.6.0 [ 0.734825] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.741719] *** VALIDATE ramfs *** [ 0.743071] *** VALIDATE hugetlbfs *** [ 0.758716] pnp: PnP ACPI init [ 0.764880] pnp: PnP ACPI: found 6 devices [ 0.785084] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.787974] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.790102] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.792081] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.794236] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.796408] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.798836] NET: Registered protocol family 2 [ 0.801033] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.809098] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.816375] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.824873] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.834412] TCP: Hash tables configured (established 65536 bind 65536) [ 0.839145] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.846420] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.853454] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.860380] NET: Registered protocol family 1 [ 0.876153] RPC: Registered named UNIX socket transport module. [ 0.878326] RPC: Registered udp transport module. [ 0.884352] RPC: Registered tcp transport module. [ 0.885538] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.893724] NET: Registered protocol family 44 [ 0.896037] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.902185] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.908405] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.914255] PCI: CLS 0 bytes, default 64 [ 0.919238] Unpacking initramfs... [ 3.290853] debug: unmapping init [mem 0xffff9f8afcc54000-0xffff9f8afffbffff] [ 3.295615] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.298423] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.303932] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.978247] Initialise system trusted keyrings [ 3.980152] Key type blacklist registered [ 3.981977] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.991329] zbud: loaded [ 3.994899] *** VALIDATE nfs *** [ 3.996131] *** VALIDATE nfs4 *** [ 3.997770] pstore: using deflate compression [ 4.001531] Platform Keyring initialized [ 4.188491] NET: Registered protocol family 38 [ 4.190291] Key type asymmetric registered [ 4.191789] Asymmetric key parser 'x509' registered [ 4.193752] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.197530] io scheduler mq-deadline registered [ 4.199618] io scheduler kyber registered [ 4.201494] io scheduler bfq registered [ 4.203630] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.206801] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.209945] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.212954] ACPI: Power Button [PWRF] [ 4.320661] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.470252] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.701956] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.820157] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.182971] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.230105] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.278918] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.284841] Non-volatile memory driver v1.3 [ 5.287320] Linux agpgart interface v0.103 [ 5.337519] virtio_blk virtio1: [vda] 133016 512-byte logical blocks (68.1 MB/64.9 MiB) [ 5.340187] vda: detected capacity change from 0 to 68104192 [ 5.368505] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.372659] vdb: detected capacity change from 0 to 1073741824 [ 5.396387] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.405189] vdc: detected capacity change from 0 to 2621440000 [ 5.418425] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.422911] vdd: detected capacity change from 0 to 2621440000 [ 5.443329] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.445514] vde: detected capacity change from 0 to 4294967296 [ 5.461762] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.464560] vdf: detected capacity change from 0 to 4294967296 [ 5.473176] libphy: Fixed MDIO Bus: probed [ 5.479763] usbcore: registered new interface driver usbserial_generic [ 5.483574] usbserial: USB Serial support registered for generic [ 5.489213] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.493435] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.494805] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.497611] mousedev: PS/2 mouse device common for all mice [ 5.499993] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.500280] rtc_cmos 00:05: RTC can wake from S4 [ 5.505177] rtc_cmos 00:05: registered as rtc0 [ 5.520054] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.521661] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.536426] hpet1: lost 1 rtc interrupts [ 5.545504] intel_pstate: CPU model not supported [ 5.548278] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.552672] hid: raw HID events driver (C) Jiri Kosina [ 5.563920] usbcore: registered new interface driver usbhid [ 5.566642] usbhid: USB HID core driver [ 5.568124] drop_monitor: Initializing network drop monitor service [ 5.569817] Initializing XFRM netlink socket [ 5.571285] NET: Registered protocol family 10 [ 5.573771] Segment Routing with IPv6 [ 5.579714] NET: Registered protocol family 17 [ 5.582331] mpls_gso: MPLS GSO support [ 5.591298] RAS: Correctable Errors collector initialized. [ 5.593644] AVX version of gcm_enc/dec engaged. [ 5.595165] AES CTR mode by8 optimization enabled [ 5.783259] sched_clock: Marking stable (5783241363, 0)->(7127444111, -1344202748) [ 5.787897] registered taskstats version 1 [ 5.789905] Loading compiled-in X.509 certificates [ 5.792677] zswap: loaded using pool lzo/zbud [ 5.846300] Key type big_key registered [ 5.868653] Key type encrypted registered [ 5.870066] ima: No TPM chip found, activating TPM-bypass! [ 5.876247] ima: Allocated hash algorithm: sha1 [ 5.877793] ima: No architecture policies found [ 5.879435] evm: Initialising EVM extended attributes: [ 5.881174] evm: security.selinux [ 5.881940] evm: security.ima [ 5.882988] evm: security.capability [ 5.884167] evm: HMAC attrs: 0x1 [ 5.886912] rtc_cmos 00:05: setting system clock to 2025-08-05 14:48:35 UTC (1754405315) [ 5.900118] debug: unmapping init [mem 0xffffffff96003000-0xffffffff961fffff] [ 5.902695] debug: unmapping init [mem 0xffffffff94d82000-0xffffffff95058fff] [ 5.910152] Write protecting the kernel read-only data: 28672k [ 5.914335] debug: unmapping init [mem 0xffffffff93403000-0xffffffff935fffff] [ 5.919512] debug: unmapping init [mem 0xffffffff93d14000-0xffffffff93dfffff] [ 5.967393] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.980513] systemd[1]: Detected virtualization kvm. [ 5.981729] systemd[1]: Detected architecture x86-64. [ 5.983522] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.025131] systemd[1]: No hostname configured. [ 6.027222] systemd[1]: Set hostname to . [ 6.029263] random: systemd: uninitialized urandom read (16 bytes read) [ 6.032990] systemd[1]: Initializing machine ID from random generator. [ 6.163277] random: ln: uninitialized urandom read (6 bytes read) [ 6.349777] random: systemd: uninitialized urandom read (16 bytes read) [ 6.355385] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 6.361910] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 6.369668] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.175368] device-mapper: uevent: version 1.0.3 [ 8.179597] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK [0[ 9.770458] virtio_net virtio0 ens2: renamed from eth0 m] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.057697] random: fast init done [ 10.385415] scsi host0: ata_piix [ 10.451240] scsi host1: ata_piix [ 10.452841] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.461245] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.411246] random: crng init done [ 14.414384] random: 7 urandom warning(s) missed due to ratelimiting [ 16.543402] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.685163] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.163659] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.615179] SELinux: Disabled at runtime. [ 20.686061] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.697794] systemd[1]: Detected virtualization kvm. [ 20.699825] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.195113] hrtimer: interrupt took 5102412 ns [ 22.123159] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.127715] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.140698] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.152359] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.161372] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.183497] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.227819] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ 22.631796] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 24.266362] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 25.449644] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.456298] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.358457] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.444686] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 31.383146] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 31.931313] NFS: Registering the id_resolver key type [ 31.935300] Key type id_resolver registered [ 31.936934] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg456-server login: [ 68.243796] libcfs: loading out-of-tree module taints kernel. [ 68.254670] Key type ._llcrypt registered [ 68.255653] Key type .llcrypt registered [ 68.301195] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 76.057660] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 76.688189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 76.696726] alg: No test for adler32 (adler32-zlib) [ 77.769329] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 78.119868] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 79.751790] Key type lgssc registered [ 80.442878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 88.035552] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 89.123976] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 93.249715] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 96.513142] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.848071] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 105.938805] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 111.545974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 111.574731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 111.599643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 112.721451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 112.737047] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 112.782134] Lustre: lustre-MDT0000: new disk, initializing [ 112.824100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 112.837641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 114.424082] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 120.362079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 120.393957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 120.422989] Lustre: 6460:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 120.437424] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 120.439215] Lustre: Skipped 1 previous similar message [ 120.478928] Lustre: lustre-MDT0001: new disk, initializing [ 120.501481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 120.511963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 120.515895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 121.949872] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 124.273459] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 127.448453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 127.472888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 127.542661] Lustre: lustre-OST0000: new disk, initializing [ 127.543964] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 127.556602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 129.398152] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 134.635369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 134.638658] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 134.661994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 135.020651] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 135.058351] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 135.101861] Lustre: lustre-OST0001: new disk, initializing [ 135.103620] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 135.124574] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 137.243674] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 143.604675] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 144.364619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 144.367771] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 144.397289] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 148.400557] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 155.627455] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing check_logdir /tmp/testlogs/ [ 157.407996] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing yml_node [ 159.360510] Lustre: DEBUG MARKER: Client: 2.16.57.28 [ 160.447775] Lustre: DEBUG MARKER: MDS: 2.16.57.28 [ 161.463407] Lustre: DEBUG MARKER: OSS: 2.16.57.28 [ 162.108050] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Aug 5 10:51:12 EDT 2025 [ 168.924485] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 169.509387] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 182.240253] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 182.244118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 182.250801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 185.314589] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 185.314621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 185.321687] Lustre: Skipped 2 previous similar messages [ 185.326337] Lustre: Skipped 3 previous similar messages [ 187.521464] LustreError: 12124:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 187.552594] Lustre: server umount lustre-MDT0000 complete [ 189.014246] LustreError: 6454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754405498 with bad export cookie 13315700795167168898 [ 189.015750] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 189.020602] LustreError: 6454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 190.431721] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 190.432264] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 190.437800] Lustre: Skipped 1 previous similar message [ 190.445642] Lustre: Skipped 1 previous similar message [ 194.464582] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 194.467344] Lustre: Skipped 1 previous similar message [ 195.142191] LustreError: 12326:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 195.144310] LustreError: 12326:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 195.251097] Lustre: server umount lustre-MDT0001 complete [ 196.898100] LustreError: 12527:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 196.901239] LustreError: 12527:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 196.922093] Lustre: server umount lustre-OST0000 complete [ 198.507778] Lustre: server umount lustre-OST0001 complete [ 201.051214] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 204.607773] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 208.480771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 210.970022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 213.170899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 215.518138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 218.401268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 218.433684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 218.541498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 218.556447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 218.596090] Lustre: lustre-MDT0000: new disk, initializing [ 218.636849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 218.649217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 220.223685] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 225.008308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 225.042502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 225.095006] Lustre: 14893:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 225.230161] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 226.870154] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 230.380361] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 231.803776] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 234.440922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 234.468749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 234.589269] Lustre: lustre-OST0000: new disk, initializing [ 234.590841] Lustre: Skipped 1 previous similar message [ 234.593251] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 234.595944] Lustre: Skipped 2 previous similar messages [ 234.639239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 234.643518] Lustre: Skipped 1 previous similar message [ 236.158057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 236.160677] Lustre: Skipped 1 previous similar message [ 236.163453] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 236.200100] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 236.806206] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 240.258512] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 241.893132] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 241.979074] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 243.549123] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 243.647828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 246.755084] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 246.755497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 246.756586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 246.758099] LustreError: Skipped 1 previous similar message [ 246.763918] Lustre: Skipped 1 previous similar message [ 250.412163] LustreError: 17799:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 250.414018] LustreError: 17799:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 250.478664] Lustre: server umount lustre-OST0000 complete [ 251.872690] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 251.878044] Lustre: Skipped 1 previous similar message [ 256.992366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 256.994880] Lustre: Skipped 4 previous similar messages [ 266.207502] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 266.318550] LustreError: 18001:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 266.333931] LustreError: 18001:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 266.435176] Lustre: server umount lustre-MDT0000 complete [ 267.241269] LustreError: 14905:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 267.248274] LustreError: 14905:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 269.916705] LustreError: 14884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754405579 with bad export cookie 13315700795167170431 [ 269.920234] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 269.935829] LustreError: 14884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 270.447853] Lustre: server umount lustre-MDT0001 complete [ 276.518098] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 10:53:05 (1754405585) [ 296.154716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 296.853775] LustreError: 19339:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 296.963838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 301.956889] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 302.051096] LustreError: 19339:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.177671] LustreError: 19339:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 312.290561] LustreError: 19339:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 312.891119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 318.618795] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 324.744782] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 328.109667] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 329.974684] Lustre: Failing over lustre-MDT0000 [ 330.101296] LustreError: 20669:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 330.104271] LustreError: 20669:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 330.247284] Lustre: server umount lustre-MDT0000 complete [ 333.283407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.294286] LustreError: 19333:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.317234] LustreError: 19333:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 347.718156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 347.787322] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 347.961584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 347.965467] Lustre: Skipped 1 previous similar message [ 350.978580] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 353.249350] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 353.253785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 353.263888] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 356.361870] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 363.488225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 363.494569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 363.503368] Lustre: Skipped 1 previous similar message [ 363.507658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 363.510413] Lustre: Skipped 2 previous similar messages [ 365.598033] LustreError: 21647:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 365.601263] LustreError: 21647:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 365.701951] Lustre: server umount lustre-MDT0000 complete [ 368.244438] LustreError: 19318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754405677 with bad export cookie 13315700795167171845 [ 368.277166] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 368.540238] Lustre: server umount lustre-MDT0001 complete [ 374.243540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 374.465768] LustreError: 22324:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.476241] LustreError: 22324:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 376.888658] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 382.894912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 385.762159] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 389.737843] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 392.500370] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 394.135391] Lustre: Failing over lustre-MDT0000 [ 394.324060] Lustre: server umount lustre-MDT0000 complete [ 407.522212] LustreError: 22318:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 407.535486] LustreError: 22318:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 411.423739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.510903] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 415.168673] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 416.738995] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 416.752557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 416.759905] Lustre: Skipped 1 previous similar message [ 416.773783] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 421.692143] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 426.981602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 426.983838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 426.990183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 426.997169] Lustre: Skipped 4 previous similar messages [ 426.999191] Lustre: Skipped 4 previous similar messages [ 439.263154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 439.351333] LustreError: 24625:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 439.353987] LustreError: 24625:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 439.423907] Lustre: server umount lustre-MDT0000 complete [ 441.923693] LustreError: 22305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754405751 with bad export cookie 13315700795167172776 [ 441.925691] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 441.934095] LustreError: 22305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 442.186214] Lustre: server umount lustre-MDT0001 complete [ 448.266127] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 10:55:57 (1754405757) [ 458.046357] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 463.008201] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 468.863807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 473.721525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 478.159705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 482.266414] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 488.325886] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 495.155862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 495.219562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.420656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 495.454697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 495.512436] Lustre: lustre-MDT0000: new disk, initializing [ 495.590604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 495.593516] Lustre: Skipped 3 previous similar messages [ 495.609116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 498.068359] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 505.699843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 505.760742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 505.838927] Lustre: 28275:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 506.092700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 509.143988] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 514.833897] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 517.253315] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 521.964533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 522.020926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 523.295847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 523.347673] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 523.389348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 523.398720] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 526.134430] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 530.949570] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 534.347597] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 548.832119] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 548.838683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 548.849677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 548.854238] Lustre: Skipped 6 previous similar messages [ 554.202231] LustreError: 30916:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 554.204214] LustreError: 30916:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 554.322234] Lustre: server umount lustre-OST0000 complete [ 557.538255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 557.541477] LustreError: Skipped 1 previous similar message [ 571.359137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 571.506729] Lustre: server umount lustre-MDT0000 complete [ 573.525735] LustreError: 28269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754405883 with bad export cookie 13315700795167173175 [ 573.532114] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 573.533296] LustreError: 28269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 573.788599] Lustre: server umount lustre-MDT0001 complete [ 580.332594] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 582.422682] Key type lgssc unregistered [ 582.658453] LNet: 31847:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 582.666188] LNetError: 31847:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 582.681470] LNet: Removed LNI 192.168.204.156@tcp [ 583.218221] Key type .llcrypt unregistered [ 583.220888] Key type ._llcrypt unregistered [ 598.206806] Key type ._llcrypt registered [ 598.208499] Key type .llcrypt registered [ 598.302062] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 598.828302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 598.895192] alg: No test for adler32 (adler32-zlib) [ 599.839813] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 599.966174] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 601.583771] Key type lgssc registered [ 602.324874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 607.725907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 607.745418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 609.071848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 611.674939] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 617.956311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 618.180758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 620.974830] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 624.679523] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 627.236528] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 631.565592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 631.775689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 633.856981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 635.149369] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 638.741519] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 643.063526] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 651.283731] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 10:59:21 (1754405961) [ 652.326826] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 653.434079] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 657.562984] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 10:59:27 (1754405967) [ 663.205876] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 10:59:32 (1754405972) [ 663.869145] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 669.330354] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 669.334659] Lustre: Skipped 2 previous similar messages [ 670.634577] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 10:59:40 (1754405980) [ 673.368167] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 673.370268] Lustre: Skipped 3 previous similar messages [ 685.275878] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 685.279977] Lustre: Skipped 6 previous similar messages [ 687.159034] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 10:59:56 (1754405996) [ 688.931221] Lustre: *** cfs_fail_loc=131b, val=2*** [ 691.417946] Lustre: *** cfs_fail_loc=131b, val=2*** [ 696.415673] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 11:00:06 (1754406006) [ 703.574562] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 703.578280] Lustre: Skipped 7 previous similar messages [ 704.584950] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 11:00:14 (1754406014) [ 714.725451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 714.725987] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 714.729132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 714.734240] Lustre: Skipped 1 previous similar message [ 718.576443] LustreError: 39755:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 718.690304] Lustre: server umount lustre-OST0000 complete [ 720.865950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 720.870428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 720.877942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 720.881971] Lustre: Skipped 1 previous similar message [ 724.960834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 724.966931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 724.976428] Lustre: Skipped 1 previous similar message [ 730.082712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 730.085207] Lustre: Skipped 1 previous similar message [ 735.200358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 735.200458] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 735.206951] Lustre: Skipped 1 previous similar message [ 735.273044] LustreError: 39956:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 735.277923] LustreError: 39956:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 735.336698] Lustre: server umount lustre-MDT0000 complete [ 737.799833] LustreError: 33191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754406047 with bad export cookie 16409785067431815477 [ 737.805647] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 737.816757] LustreError: 33191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 737.921821] LustreError: 40158:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 737.931558] LustreError: 40158:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 738.063933] Lustre: server umount lustre-MDT0001 complete [ 744.673651] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 746.606475] Key type lgssc unregistered [ 746.808843] LNet: 40685:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.815785] LNetError: 40685:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.826150] LNet: Removed LNI 192.168.204.156@tcp [ 747.401878] Key type .llcrypt unregistered [ 747.404126] Key type ._llcrypt unregistered [ 759.019024] Key type ._llcrypt registered [ 759.020585] Key type .llcrypt registered [ 759.082455] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 768.607636] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 769.148762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 769.204138] alg: No test for adler32 (adler32-zlib) [ 770.146683] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 770.292373] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 771.938926] Key type lgssc registered [ 772.695216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 778.554831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 782.743164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 786.244206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 789.925215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 794.187423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 794.231409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 794.243384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 795.371661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 795.388844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 795.435716] Lustre: lustre-MDT0000: new disk, initializing [ 795.491697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 795.506521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 797.764704] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 803.974509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 804.009071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 804.057736] Lustre: 43789:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 804.094128] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 804.097400] Lustre: Skipped 1 previous similar message [ 804.141503] Lustre: lustre-MDT0001: new disk, initializing [ 804.191583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 804.207919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 804.215733] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 806.196579] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 810.634119] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 812.765543] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 816.617855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 816.652623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 816.791963] Lustre: lustre-OST0000: new disk, initializing [ 816.798274] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 816.862303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 819.748222] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 820.236532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 820.246230] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 820.304955] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 824.200536] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 826.457793] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 826.571503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 828.573282] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 828.684714] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 830.437496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 830.437680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 830.450992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 830.453807] Lustre: Skipped 1 previous similar message [ 835.557518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 835.559928] Lustre: Skipped 1 previous similar message [ 835.762133] LustreError: 46696:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 835.856429] Lustre: server umount lustre-OST0000 complete [ 840.162952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 840.166658] LustreError: Skipped 1 previous similar message [ 840.169082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 840.175279] Lustre: Skipped 1 previous similar message [ 840.179452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 843.972592] LustreError: 46897:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 843.974584] LustreError: 46897:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 844.021380] Lustre: server umount lustre-MDT0000 complete [ 845.701521] LustreError: 45522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754406155 with bad export cookie 16496233174830537238 [ 845.705525] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 845.706857] LustreError: 45522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 845.774305] LustreError: 47098:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 845.777618] LustreError: 47098:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 845.887546] Lustre: server umount lustre-MDT0001 complete [ 850.651858] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 11:02:40 (1754406160) [ 854.223855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 854.509542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 856.636624] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 859.618651] LustreError: 47818:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 859.625340] LustreError: 47818:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 861.242150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 861.454632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 863.450129] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 866.166073] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 867.933879] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 872.000801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 872.205992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 875.953445] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 879.408113] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 891.626708] LustreError: 50014:0:(llog.c:430:llog_init_handle()) llog type is not specified! [ 892.090155] Lustre: Modifying parameter general.timeout=1 in log params [ 893.407560] Lustre: Modifying parameter general.timeout=4 in log params [ 893.410147] Lustre: Skipped 2 previous similar messages [ 895.709730] Lustre: Modifying parameter general.timeout=9 in log params [ 895.713272] Lustre: Skipped 4 previous similar messages [ 900.084364] Lustre: Modifying parameter general.timeout=18 in log params [ 900.086417] Lustre: Skipped 8 previous similar messages [ 908.091563] Lustre: Modifying parameter general.timeout=34 in log params [ 908.094097] Lustre: Skipped 15 previous similar messages [ 924.138184] Lustre: Modifying parameter general.timeout=66 in log params [ 924.140897] Lustre: Skipped 31 previous similar messages [ 956.440051] Lustre: Disabling parameter general.timeout= in log params [ 956.442479] Lustre: Skipped 63 previous similar messages [ 957.887480] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 11:04:27 (1754406267) [ 959.206176] Lustre: 56927:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 960.968473] Lustre: 57123:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 960.974972] Lustre: 57123:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 1 previous similar message [ 963.528254] Lustre: 57434:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 963.533280] Lustre: 57434:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 2 previous similar messages [ 975.538640] Lustre: 58775:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 975.547858] LustreError: 58775:0:(mgs_handler.c:1119:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 977.325868] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 11:04:47 (1754406287) [ 984.034337] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 984.035558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 984.039632] Lustre: Skipped 3 previous similar messages [ 984.042557] Lustre: Skipped 2 previous similar messages [ 988.487815] LustreError: 59471:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 988.490824] LustreError: 59471:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 988.534498] Lustre: server umount lustre-MDT0000 complete [ 989.153185] LustreError: 47817:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 989.161283] LustreError: 47817:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 990.420656] LustreError: 47798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754406300 with bad export cookie 16496233174830538120 [ 990.423516] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 990.425724] LustreError: 47798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 990.624970] Lustre: server umount lustre-MDT0001 complete [ 1002.869587] LustreError: 59875:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1002.872873] LustreError: 59875:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1002.972764] Lustre: server umount lustre-OST0000 complete [ 1006.802250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1008.600050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1010.419420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1012.204989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1015.644616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1015.781604] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1015.912312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1017.717264] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1021.893767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1021.958193] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1021.973052] Lustre: 61323:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1023.885788] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1026.408794] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1028.037895] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1031.020818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1031.111232] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1033.854909] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1036.377118] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1042.476487] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1042.481519] Lustre: Skipped 23 previous similar messages [ 1042.503144] Lustre: 63536:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1048.035646] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1048.041877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1048.054668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1048.060382] Lustre: Skipped 2 previous similar messages [ 1051.564100] LustreError: 63792:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1051.567331] LustreError: 63792:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1051.655552] Lustre: server umount lustre-OST0000 complete [ 1058.273110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1058.274874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1058.275674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1058.278798] LustreError: Skipped 1 previous similar message [ 1058.289082] Lustre: Skipped 2 previous similar messages [ 1059.551236] Lustre: server umount lustre-MDT0000 complete [ 1061.270269] LustreError: 61314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754406370 with bad export cookie 16496233174830540626 [ 1061.275675] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1061.465815] Lustre: server umount lustre-MDT0001 complete [ 1066.963046] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 1068.517134] Key type lgssc unregistered [ 1068.693527] LNet: 64725:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.698398] LNetError: 64725:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1068.712323] LNet: Removed LNI 192.168.204.156@tcp [ 1069.234900] Key type .llcrypt unregistered [ 1069.236493] Key type ._llcrypt unregistered [ 1081.609173] Key type ._llcrypt registered [ 1081.610411] Key type .llcrypt registered [ 1081.666797] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 1082.059569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1082.120370] alg: No test for adler32 (adler32-zlib) [ 1083.034830] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 1083.138931] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 1084.751161] Key type lgssc registered [ 1085.357721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1089.154434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1089.172152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1090.449969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1092.456410] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1096.891635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1097.197725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1098.991735] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1101.364623] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1102.966540] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1105.963538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1106.255428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.835367] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1111.434934] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1119.663631] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1122.784659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1122.788277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.795933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1125.858392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1125.859027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1125.860348] Lustre: Skipped 1 previous similar message [ 1125.865295] Lustre: Skipped 1 previous similar message [ 1128.774887] LustreError: 68789:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1128.811155] Lustre: server umount lustre-MDT0000 complete [ 1130.276516] LustreError: 68430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754406439 with bad export cookie 15833763580292815132 [ 1130.279088] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1130.281847] LustreError: 68430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1130.352173] LustreError: 68990:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1130.353988] LustreError: 68990:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1130.436426] Lustre: server umount lustre-MDT0001 complete [ 1142.255911] LustreError: 69190:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1142.257760] LustreError: 69190:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1142.326500] Lustre: server umount lustre-OST0000 complete [ 1146.203854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1147.935261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1149.530459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1151.231844] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1154.330187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1154.479927] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1154.502808] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1154.640843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1156.556601] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1160.914798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1160.977527] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1160.992415] Lustre: 70639:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1161.242238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1163.432217] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1165.999231] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1167.470406] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1170.465803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1170.568493] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1170.751568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1173.518673] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1176.246655] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1182.386556] Lustre: 72872:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1183.434094] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1187.811996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1187.812551] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1187.818905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1187.823121] Lustre: Skipped 2 previous similar messages [ 1191.729428] LustreError: 73130:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1191.731845] LustreError: 73130:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1191.806025] Lustre: server umount lustre-OST0000 complete [ 1197.024226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1197.027751] LustreError: Skipped 1 previous similar message [ 1197.030395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1197.036431] Lustre: Skipped 1 previous similar message [ 1197.039355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1199.697940] LustreError: 73331:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1199.701849] LustreError: 73331:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1199.754483] Lustre: server umount lustre-MDT0000 complete [ 1201.762232] LustreError: 71363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754406511 with bad export cookie 15833763580292815958 [ 1201.763641] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1201.766526] LustreError: 71363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1202.021434] Lustre: server umount lustre-MDT0001 complete [ 1207.122234] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 1208.531285] Key type lgssc unregistered [ 1208.716392] LNet: 74057:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1208.722432] LNetError: 74057:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1208.739604] LNet: Removed LNI 192.168.204.156@tcp [ 1209.152927] Key type .llcrypt unregistered [ 1209.154531] Key type ._llcrypt unregistered [ 1220.980650] Key type ._llcrypt registered [ 1220.981908] Key type .llcrypt registered [ 1221.039959] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 1221.466111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1221.530165] alg: No test for adler32 (adler32-zlib) [ 1222.435557] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 1222.550932] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 1224.159280] Key type lgssc registered [ 1224.701538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1228.241633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1228.253881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1229.501831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1231.263124] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1234.858900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1235.162615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1236.976257] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1239.430458] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1241.093392] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1244.308788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1244.609693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1247.432284] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1249.839495] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1257.894442] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1258.744332] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1258.746460] Lustre: Skipped 1 previous similar message [ 1260.016994] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1260.019719] Lustre: Skipped 2 previous similar messages [ 1262.099061] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1262.101732] Lustre: Skipped 4 previous similar messages [ 1266.245591] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1266.247408] Lustre: Skipped 9 previous similar messages [ 1274.529738] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1274.533179] Lustre: Skipped 19 previous similar messages [ 1290.603537] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1290.606341] Lustre: Skipped 38 previous similar messages [ 1301.957569] Lustre: 83143:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1303.153107] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 11:10:13 (1754406613) [ 1303.822315] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1304.603196] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 11:10:14 (1754406614) [ 1306.080401] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1306.083163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1306.084622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1306.088515] Lustre: Skipped 1 previous similar message [ 1311.201610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1311.204429] Lustre: Skipped 2 previous similar messages [ 1311.916835] LustreError: 83435:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1311.984697] Lustre: server umount lustre-OST0000 complete [ 1316.321123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1316.325094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1316.325971] Lustre: Skipped 1 previous similar message [ 1319.551096] LustreError: 83639:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1319.554018] LustreError: 83639:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1319.581574] Lustre: server umount lustre-MDT0000 complete [ 1321.092517] LustreError: 75404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754406630 with bad export cookie 5874773366931831875 [ 1321.095884] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1321.096966] LustreError: 75404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1321.166777] LustreError: 83841:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1321.169549] LustreError: 83841:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1321.272154] Lustre: server umount lustre-MDT0001 complete [ 1325.859949] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 1327.171107] Key type lgssc unregistered [ 1327.310434] LNet: 84367:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1327.313365] LNetError: 84367:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1327.320610] LNet: Removed LNI 192.168.204.156@tcp [ 1327.705894] Key type .llcrypt unregistered [ 1327.707619] Key type ._llcrypt unregistered [ 1329.253130] Key type ._llcrypt registered [ 1329.254736] Key type .llcrypt registered [ 1329.301204] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1330.969668] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules [ 1331.400259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1331.449684] alg: No test for adler32 (adler32-zlib) [ 1332.334600] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 1332.435268] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 1334.031215] Key type lgssc registered [ 1334.539443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1335.066919] LustreError: 85471:0:(lu_object.c:1390:lu_context_key_register()) cfs_race id 60d sleeping [ 1343.885702] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 1347.732121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1356.392433] LustreError: 85471:0:(lu_object.c:1390:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1356.413597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1357.708809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1359.321185] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1362.858945] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 11:11:12 (1754406672) [ 1364.876605] LustreError: 86506:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1364.881358] LustreError: 86506:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1364.885407] LustreError: 86506:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 1364.962138] LustreError: 87272:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1365.070394] Lustre: server umount lustre-MDT0000 complete [ 1371.082357] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 1372.419082] Key type lgssc unregistered [ 1372.587046] LNet: 87951:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1372.590272] LNetError: 87951:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1372.599684] LNet: Removed LNI 192.168.204.156@tcp [ 1372.992857] Key type .llcrypt unregistered [ 1372.994381] Key type ._llcrypt unregistered [ 1381.300414] Key type ._llcrypt registered [ 1381.301534] Key type .llcrypt registered [ 1381.346701] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 1381.740344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1381.780645] alg: No test for adler32 (adler32-zlib) [ 1382.656211] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 1382.756425] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 1384.359285] Key type lgssc registered [ 1384.865956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1388.066224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1388.077269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1389.320815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1390.907585] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1394.566257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1394.883256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1396.486576] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1398.765260] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1400.284820] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1403.077918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1403.348190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1405.706365] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1408.047861] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1438.885621] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 11:12:28 (1754406748) [ 1439.641531] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1440.464536] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 11:12:30 (1754406750) [ 1444.321581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1444.322906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1444.328296] Lustre: Skipped 2 previous similar messages [ 1444.330868] Lustre: Skipped 1 previous similar message [ 1447.938506] LustreError: 92010:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1447.976826] Lustre: server umount lustre-MDT0000 complete [ 1449.440063] LustreError: 89121:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1449.447370] LustreError: 89121:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 1449.524346] LustreError: 89102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754406759 with bad export cookie 7911065410179827417 [ 1449.527754] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1449.529780] LustreError: 89102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1449.602524] LustreError: 92211:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1449.604549] LustreError: 92211:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1449.705371] Lustre: server umount lustre-MDT0001 complete [ 1461.621708] LustreError: 92413:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1461.624721] LustreError: 92413:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1461.823895] Lustre: server umount lustre-OST0000 complete [ 1466.868913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1467.094118] LustreError: 93042:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1467.182415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1468.871707] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1472.480223] LustreError: 93042:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1472.767239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1473.094922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1474.797573] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1477.250341] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1478.863827] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1481.493053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1484.209362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1484.243288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1484.321816] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1484.325986] LustreError: 93031:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1484.339904] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1484.344424] LustreError: 94891:0:(tgt_mount.c:2419:server_fill_super()) Unable to start targets: -98 [ 1484.347820] LustreError: 94891:0:(tgt_mount.c:1913:server_put_super()) no obd lustre-OST0000 [ 1484.350874] LustreError: 94891:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1484.412117] Lustre: server umount lustre-OST0000 complete [ 1484.413965] LustreError: 94891:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1488.015971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1488.113919] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1488.117797] LustreError: 93031:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1488.122488] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1488.128691] LustreError: 95353:0:(tgt_mount.c:2419:server_fill_super()) Unable to start targets: -98 [ 1488.133829] LustreError: 95353:0:(tgt_mount.c:1913:server_put_super()) no obd lustre-OST0000 [ 1488.138045] LustreError: 95353:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1488.194377] Lustre: server umount lustre-OST0000 complete [ 1488.196442] LustreError: 95353:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1489.454890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1492.356549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1492.465962] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1492.468711] Lustre: Found index 0 for lustre-OST0000, updating log [ 1492.471092] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1492.508844] Lustre: lustre-OST0000: new disk, initializing [ 1492.511849] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1492.632889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1494.955096] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1496.407588] Lustre: Failing over lustre-OST0000 [ 1496.424912] LustreError: 96424:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1496.427503] LustreError: 96424:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1496.491183] Lustre: server umount lustre-OST0000 complete [ 1498.593155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1498.596828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1498.602783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1498.605245] Lustre: Skipped 1 previous similar message [ 1503.201591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1503.203566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1504.069259] LustreError: 96624:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1504.071308] LustreError: 96624:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1504.111648] Lustre: server umount lustre-MDT0000 complete [ 1505.633500] LustreError: 93023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754406815 with bad export cookie 7911065410179830147 [ 1505.636598] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1505.638257] LustreError: 93023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1505.787234] Lustre: server umount lustre-MDT0001 complete [ 1509.164950] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 11:13:39 (1754406819) [ 1515.418397] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 1519.364121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1519.531710] LustreError: 98190:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1519.571439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1521.126389] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1524.611154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1524.703916] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1524.706648] Lustre: Skipped 1 previous similar message [ 1526.581343] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1527.755681] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1530.349546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1532.828177] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1536.172327] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1536.202575] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1536.232103] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1536.272073] Lustre: lustre-OST0001: new disk, initializing [ 1536.274869] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1538.660381] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1541.129627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1541.133108] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1541.166176] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1547.429556] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1552.525075] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1560.089779] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.56@tcp (stopping) [ 1560.546807] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1560.552550] Lustre: Skipped 2 previous similar messages [ 1565.247363] LustreError: 101403:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 1565.249965] LustreError: 101403:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1565.282889] Lustre: server umount lustre-MDT0001 complete [ 1565.667088] LustreError: 98894:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1565.674232] LustreError: 98894:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1566.462573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1569.198285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1569.258343] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1569.261188] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1569.281819] Lustre: 98179:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1569.598224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1569.601862] Lustre: Skipped 3 previous similar messages [ 1570.406948] LustreError: 102031:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1570.410922] LustreError: 102031:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1570.415494] LustreError: 98181:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1570.455088] LustreError: 102042:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.204.156@tcp: cfg command failed: rc = -17 [ 1570.458845] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1570.458845] [ 1571.202644] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1574.888958] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1575.172140] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 11:14:45 (1754406885) [ 1582.148479] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 1590.052534] Lustre: 103997:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1600.307296] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1601.804685] Lustre: 105130:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1661.921361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1661.921484] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1661.922204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1661.922210] Lustre: Skipped 4 previous similar messages [ 1661.935128] Lustre: Skipped 2 previous similar messages [ 1664.726050] LustreError: 105585:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1664.728415] LustreError: 105585:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1664.763236] Lustre: server umount lustre-MDT0000 complete [ 1666.345964] LustreError: 99286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754406975 with bad export cookie 7911065410179831015 [ 1666.347424] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1666.350550] LustreError: 99286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1666.516105] Lustre: server umount lustre-MDT0001 complete [ 1678.417561] Lustre: server umount lustre-OST0000 complete [ 1690.306457] Lustre: server umount lustre-OST0001 complete [ 1692.705887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1696.662036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1697.047603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1700.472272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1704.128064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1704.534588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1711.671482] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 1715.730403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1715.738674] Lustre: lustre-MDT0000: reset Object Index mappings [ 1715.926184] LustreError: 109058:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1715.963632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1717.490956] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1721.088798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1721.097077] Lustre: lustre-MDT0001: reset Object Index mappings [ 1723.114409] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1724.447789] Lustre: 110160:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1727.503235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1730.416479] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1730.983440] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 1734.003630] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1735.985692] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 1736.616548] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1741.099263] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 1741.100436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 1745.613383] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1747.129762] Lustre: 112000:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1771.696834] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 11:18:01 (1754407081) [ 1800.673264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.675818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1800.678940] Lustre: Skipped 3 previous similar messages [ 1800.681978] Lustre: Skipped 4 previous similar messages [ 1806.359598] LustreError: 112353:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1806.362042] LustreError: 112353:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 1806.397144] Lustre: server umount lustre-MDT0000 complete [ 1807.850798] LustreError: 109785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754407117 with bad export cookie 7911065410179877810 [ 1807.856150] LustreError: 109785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1807.858389] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1810.912215] LustreError: 111327:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1810.917661] LustreError: 111327:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1814.257837] Lustre: server umount lustre-MDT0001 complete [ 1830.367252] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1830.418459] Lustre: server umount lustre-OST0000 complete [ 1846.241235] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1846.353467] Lustre: server umount lustre-OST0001 complete [ 1848.943161] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 1852.702788] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 1856.740257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1859.484198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1861.998126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1864.910350] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1868.071268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1868.104392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1868.245523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1868.272164] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1868.317851] Lustre: lustre-MDT0000: new disk, initializing [ 1868.378987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1868.381751] Lustre: Skipped 3 previous similar messages [ 1868.391903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1870.351503] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1873.870139] LustreError: 115690:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1873.873044] LustreError: 115690:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 1873.974882] Lustre: server umount lustre-MDT0000 complete [ 1875.245757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1878.319867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1878.365120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1878.515406] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1878.523652] Lustre: 116212:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1878.530611] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1880.389443] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1882.709502] Lustre: server umount lustre-MDT0000 complete [ 1886.696427] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 11:19:56 (1754407196) [ 1887.446475] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1888.406874] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 11:19:58 (1754407198) [ 1895.724723] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 1900.021868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1901.892418] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1905.929964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1905.968251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1906.021912] Lustre: 118217:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1906.057374] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1906.059621] Lustre: Skipped 1 previous similar message [ 1906.101882] Lustre: lustre-MDT0001: new disk, initializing [ 1906.176675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1906.182684] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1907.986611] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1910.745574] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1913.547562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1913.580712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1913.708766] Lustre: lustre-OST0000: new disk, initializing [ 1913.712916] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1914.160321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1914.165673] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1914.225590] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1916.155950] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1921.117865] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1921.149203] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1923.143085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1923.146978] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1923.191389] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1923.473776] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1928.331831] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1929.894774] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2086.553440] Lustre: 126956:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2086.557216] Lustre: 126956:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2086.559415] Lustre: 126956:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2086.561847] Lustre: 126956:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2086.564604] Lustre: 126956:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2086.567413] Lustre: 126956:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2184.378684] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:24:54 (1754407494) [ 2197.984327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2197.987578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2197.992319] Lustre: Skipped 3 previous similar messages [ 2197.995878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2197.998084] Lustre: Skipped 8 previous similar messages [ 2202.125722] LustreError: 127164:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2202.128344] LustreError: 127164:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2202.210732] Lustre: server umount lustre-MDT0000 complete [ 2203.825892] LustreError: 118207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754407513 with bad export cookie 7911065410179934174 [ 2203.831184] LustreError: 118207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2203.832627] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2205.153020] LustreError: 118224:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2205.159424] LustreError: 118224:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2210.130980] Lustre: server umount lustre-MDT0001 complete [ 2226.143197] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2226.180823] Lustre: server umount lustre-OST0000 complete [ 2233.854537] Lustre: server umount lustre-OST0001 complete [ 2236.309518] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 2240.077385] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 2244.375662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2246.922995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2249.300490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2251.749272] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2254.428407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2254.454740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.550651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2254.565960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2254.568274] Lustre: Skipped 1 previous similar message [ 2254.607248] Lustre: lustre-MDT0000: new disk, initializing [ 2254.608581] Lustre: Skipped 1 previous similar message [ 2254.641536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2254.644664] Lustre: Skipped 5 previous similar messages [ 2254.653308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2256.266884] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2260.732236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2260.756716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2260.794305] Lustre: 129946:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2260.915125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2262.255503] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2265.443111] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2266.773470] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2269.162909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2269.185277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2270.528766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2270.573308] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2271.291928] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2274.459253] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2277.870404] Lustre: lustre-MDD0000: changelog on [ 2278.811106] Lustre: lustre-MDD0001: changelog on [ 2674.549832] Lustre: 129953:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2703.477246] Lustre: lustre-MDD0001: changelog off [ 2704.730593] Lustre: lustre-MDD0000: changelog off [ 2711.011779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2711.012659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2711.014775] Lustre: Skipped 5 previous similar messages [ 2711.015747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2711.016935] LustreError: Skipped 1 previous similar message [ 2711.023017] Lustre: Skipped 9 previous similar messages [ 2712.366122] LustreError: 139477:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2712.369355] LustreError: 139477:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2712.445975] Lustre: server umount lustre-OST0000 complete [ 2719.817694] Lustre: server umount lustre-MDT0000 complete [ 2721.216620] LustreError: 129937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408030 with bad export cookie 7911065410180735408 [ 2721.219125] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2721.220069] LustreError: 129937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2721.395627] Lustre: server umount lustre-MDT0001 complete [ 2725.403571] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 2726.652101] Key type lgssc unregistered [ 2726.807389] LNet: 140405:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2726.809293] LNetError: 140405:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2726.819425] LNet: Removed LNI 192.168.204.156@tcp [ 2727.104104] Key type .llcrypt unregistered [ 2727.105293] Key type ._llcrypt unregistered [ 2733.343266] Key type ._llcrypt registered [ 2733.344370] Key type .llcrypt registered [ 2733.380557] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 2739.299775] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 2739.626583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2739.654595] alg: No test for adler32 (adler32-zlib) [ 2740.505772] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 2740.580897] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 2742.159175] Key type lgssc registered [ 2742.524496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2745.608893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2748.659807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2753.168939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2759.339208] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2765.389585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2765.455850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2765.471764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2766.696487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2766.737701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2766.839973] Lustre: lustre-MDT0000: new disk, initializing [ 2766.989608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2767.010383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2770.904446] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2781.325391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2781.417044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2781.542496] Lustre: 143542:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2781.639302] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2781.642242] Lustre: Skipped 1 previous similar message [ 2781.697715] Lustre: lustre-MDT0001: new disk, initializing [ 2781.777954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2781.811666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2781.815804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2784.918596] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2789.156933] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2790.762934] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2793.457612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2793.488127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2793.589392] Lustre: lustre-OST0000: new disk, initializing [ 2793.592122] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2793.626236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2795.819874] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2799.394337] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2800.124940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2800.128467] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2800.163049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2806.873630] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_module obdecho/obdecho [ 2807.577753] LustreError: 146140:0:(echo_client.c:670:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2807.580513] LustreError: 146140:0:(obd_config.c:779:class_setup()) setup ec failed (-22) [ 2808.215396] LustreError: 146236:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 2812.384350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2812.387783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2812.395097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2815.457067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2815.458065] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2818.050922] LustreError: 146577:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2818.086400] Lustre: server umount lustre-MDT0000 complete [ 2819.568360] LustreError: 143534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408129 with bad export cookie 5305346229817800651 [ 2819.571274] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2819.572930] LustreError: 143534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2819.641814] LustreError: 146779:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2819.644763] LustreError: 146779:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2819.742535] Lustre: server umount lustre-MDT0001 complete [ 2830.515803] LustreError: 146979:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2830.517888] LustreError: 146979:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2830.604602] Lustre: server umount lustre-OST0000 complete [ 2838.983720] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 2842.892926] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 2846.888273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2849.604708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2851.940199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2854.449418] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2857.210979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2857.238451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2857.328507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2857.341956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2857.384207] Lustre: lustre-MDT0000: new disk, initializing [ 2857.419282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2857.428250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2858.843686] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2863.077584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2863.099454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2863.135463] Lustre: 149910:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2863.159725] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2863.161716] Lustre: Skipped 1 previous similar message [ 2863.201311] Lustre: lustre-MDT0001: new disk, initializing [ 2863.240441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2863.255355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2863.260417] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2864.602233] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2867.861762] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2869.392924] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2872.137761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2872.173071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2872.295558] Lustre: lustre-OST0000: new disk, initializing [ 2872.298105] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2872.329824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2874.468175] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2877.434880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2877.439966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2877.457078] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2878.150164] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2879.990962] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2880.083108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2881.597855] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2881.691654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2882.527847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2882.530940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2882.534071] Lustre: Skipped 1 previous similar message [ 2882.535543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2882.537521] Lustre: Skipped 2 previous similar messages [ 2885.601660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2885.602615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2885.605251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2888.495925] LustreError: 152821:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 2888.498496] LustreError: 152821:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2888.567898] Lustre: server umount lustre-OST0000 complete [ 2890.720438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2890.721064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2890.726827] Lustre: Skipped 1 previous similar message [ 2890.728647] Lustre: Skipped 1 previous similar message [ 2896.126927] LustreError: 153022:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2896.129773] LustreError: 153022:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2896.165681] Lustre: server umount lustre-MDT0000 complete [ 2897.696433] LustreError: 149903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408207 with bad export cookie 5305346229817801610 [ 2897.699025] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2897.701394] LustreError: 149903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2897.861371] Lustre: server umount lustre-MDT0001 complete [ 2899.278324] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 11:36:49 (1754408209) [ 2899.862094] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2900.494949] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 11:36:50 (1754408210) [ 2903.303031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2903.463899] LustreError: 153895:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2903.473308] LustreError: 153895:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2903.496690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2904.962521] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2908.084371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2909.610978] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2911.596526] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2912.999653] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2915.672807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2918.101966] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2920.134179] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2933.109323] LustreError: 156140:0:(ldlm_resource.c:981:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200000bd0:0x1:0x0].0x50053930 (000000003dc0c6f4) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2933.727860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2933.730351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2933.734951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2933.737230] Lustre: Skipped 2 previous similar messages [ 2939.201584] LustreError: 156140:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2939.203835] LustreError: 156140:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2939.233929] Lustre: server umount lustre-MDT0000 complete [ 2939.361120] LustreError: 154599:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2939.368482] LustreError: 154599:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2940.606664] LustreError: 153877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408250 with bad export cookie 5305346229817802478 [ 2940.611536] LustreError: 153877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2940.612287] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2940.945985] LustreError: 153889:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2944.479846] LustreError: 156030:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2946.859698] Lustre: server umount lustre-MDT0001 complete [ 2949.471945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2954.574788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2954.666428] LustreError: 157401:0:(llog.c:1602:llog_backup()) MGC192.168.204.156@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2954.669547] Lustre: 157401:0:(mgc_request_server.c:630:mgc_llog_local_copy()) MGC192.168.204.156@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2970.514273] LustreError: 157408:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2970.520127] LustreError: 157408:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2980.880846] LustreError: 157408:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2980.887175] LustreError: 157408:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 7 previous similar messages [ 2997.265067] LustreError: 157426:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2997.270643] LustreError: 157426:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 14 previous similar messages [ 3015.647539] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3015.650331] Lustre: 157401:0:(mgc_request.c:1876:mgc_process_log()) MGC192.168.204.156@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3031.008240] LustreError: 157406:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3031.014965] LustreError: 157406:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 19 previous similar messages [ 3042.783388] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3042.786339] Lustre: 153887:0:(mgc_request.c:1876:mgc_process_log()) MGC192.168.204.156@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3056.618787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3056.622022] Lustre: Skipped 2 previous similar messages [ 3058.046209] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3060.709603] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 3061.027377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3062.815339] LustreError: 153887:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3086.307283] Lustre: Evicted from MGS (at 192.168.204.156@tcp) after server handle changed from 0x0 to 0x49a060497ebf9855 [ 3086.311407] Lustre: MGC192.168.204.156@tcp: Connection restored to (at 0@lo) [ 3087.991844] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3089.996589] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3090.604155] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3091.429537] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 3091.844768] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3093.473513] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 1 sec [ 3101.668112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3101.669229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3101.676149] Lustre: Skipped 3 previous similar messages [ 3101.678617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3101.680875] Lustre: Skipped 10 previous similar messages [ 3105.083815] LustreError: 158877:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3105.086775] LustreError: 158877:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3105.115706] Lustre: server umount lustre-MDT0000 complete [ 3106.372312] LustreError: 158878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408415 with bad export cookie 5305346229817808981 [ 3106.375476] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3106.377865] LustreError: 158878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3106.541542] Lustre: server umount lustre-MDT0001 complete [ 3118.200063] Lustre: server umount lustre-OST0000 complete [ 3125.099252] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 3128.471390] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 3132.055452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3134.335253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3136.351338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3138.416393] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3140.670963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3140.693957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3140.780305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3140.792225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3140.829829] Lustre: lustre-MDT0000: new disk, initializing [ 3140.855462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3140.857786] Lustre: Skipped 1 previous similar message [ 3140.863527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3142.151355] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3146.296549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3146.318357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3146.347612] Lustre: 162210:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3146.443941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3147.740868] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3150.759326] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3151.864710] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3154.030755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3154.051144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3155.944707] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3156.470382] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3156.503343] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3159.032076] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3160.461726] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3160.537291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3161.832139] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3161.902522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3166.688930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3166.689992] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3166.691025] LustreError: Skipped 1 previous similar message [ 3166.695166] Lustre: Skipped 2 previous similar messages [ 3166.697321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3166.699471] Lustre: Skipped 1 previous similar message [ 3168.557224] LustreError: 165117:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3168.558994] LustreError: 165117:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3168.617441] Lustre: server umount lustre-OST0000 complete [ 3176.020593] Lustre: server umount lustre-MDT0000 complete [ 3176.927939] LustreError: 162990:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3176.934457] LustreError: 162990:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 41 previous similar messages [ 3177.292113] LustreError: 162200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408486 with bad export cookie 5305346229817825774 [ 3177.295304] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3177.298774] LustreError: 162200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3177.428835] Lustre: server umount lustre-MDT0001 complete [ 3180.292330] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:41:30 (1754408490) [ 3182.545138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3184.050924] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3186.996989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3188.434066] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3190.167319] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3191.296314] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3193.454453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3195.307217] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3197.082221] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3216.453621] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:42:06 (1754408526) [ 3218.401593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3223.613928] Lustre: server umount lustre-OST0000 complete [ 3233.759964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3233.762342] Lustre: Skipped 9 previous similar messages [ 3239.391145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3239.420300] LustreError: 168835:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3239.422064] LustreError: 168835:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3239.448588] Lustre: server umount lustre-MDT0000 complete [ 3240.684113] LustreError: 166170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408550 with bad export cookie 5305346229817826642 [ 3240.687389] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3240.687481] LustreError: 166170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3240.826927] Lustre: server umount lustre-MDT0001 complete [ 3244.424871] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 3245.526581] Key type lgssc unregistered [ 3245.679391] LNet: 169564:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3245.684401] LNetError: 169564:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3245.698332] LNet: Removed LNI 192.168.204.156@tcp [ 3246.024471] Key type .llcrypt unregistered [ 3246.025575] Key type ._llcrypt unregistered [ 3253.671349] Key type ._llcrypt registered [ 3253.672385] Key type .llcrypt registered [ 3253.715696] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 3254.104477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3254.129917] alg: No test for adler32 (adler32-zlib) [ 3255.010069] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 3255.105270] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 3256.695164] Key type lgssc registered [ 3257.097533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3260.000315] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3260.008235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3277.535297] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3283.679239] LustreError: 170756:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.156@tcp: failed processing log, type 4: rc = -110 [ 3289.823373] LustreError: 170756:0:(llog_osd.c:243:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3289.827414] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 3289.829544] LustreError: MGC192.168.204.156@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 3289.833365] LustreError: 170756:0:(tgt_mount.c:1669:server_start_targets()) failed to start server lustre-OST0000: -22 [ 3289.836157] LustreError: 170756:0:(tgt_mount.c:2419:server_fill_super()) Unable to start targets: -22 [ 3289.838881] LustreError: 170756:0:(tgt_mount.c:1913:server_put_super()) no obd lustre-OST0000 [ 3289.841185] LustreError: 170756:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3289.887688] Lustre: server umount lustre-OST0000 complete [ 3289.889602] LustreError: 170756:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 3292.904277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3294.073134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3295.391658] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3298.333850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3298.459545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3299.873163] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3301.631503] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3302.860022] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3305.190704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3305.266358] LustreError: 172836:0:(llog_osd.c:243:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3305.270152] Lustre: 172836:0:(mgc_request_server.c:610:mgc_llog_local_copy()) MGC192.168.204.156@tcp: invalid local config log lustre-OST0000: rc = -22 [ 3305.329901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3307.369431] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3309.210184] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3314.147855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3314.150391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3314.156350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3315.169373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3315.169621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3315.171416] Lustre: Skipped 1 previous similar message [ 3315.175432] Lustre: Skipped 1 previous similar message [ 3320.288779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3320.290626] Lustre: Skipped 2 previous similar messages [ 3324.383102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3324.413856] LustreError: 173480:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3324.416739] LustreError: 173480:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3324.438591] Lustre: server umount lustre-MDT0000 complete [ 3325.408713] LustreError: 171964:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3325.414417] LustreError: 171964:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 3325.769470] LustreError: 171971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408635 with bad export cookie 3431090125501021658 [ 3325.773534] LustreError: 171971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3325.773898] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3325.842368] LustreError: 173685:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3325.844343] LustreError: 173685:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3325.935572] Lustre: server umount lustre-MDT0001 complete [ 3337.584629] LustreError: 173886:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3337.587449] LustreError: 173886:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3337.675769] Lustre: server umount lustre-OST0000 complete [ 3340.866756] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 3344.313917] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 3348.020490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3350.503686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3352.882285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3355.158589] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3357.530813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3357.555219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3357.649148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3357.662505] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3357.700436] Lustre: lustre-MDT0000: new disk, initializing [ 3357.733655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3357.742282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3359.071504] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3363.150960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3363.177595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3363.211599] Lustre: 176208:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3363.234531] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3363.236334] Lustre: Skipped 1 previous similar message [ 3363.270626] Lustre: lustre-MDT0001: new disk, initializing [ 3363.304325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3363.317686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3363.321719] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3364.632226] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3367.785028] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3369.110713] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3371.554751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3371.579777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3371.671113] Lustre: lustre-OST0000: new disk, initializing [ 3371.673289] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3371.700075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3372.664841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3372.667765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3372.701388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3373.606818] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3376.608625] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3377.968165] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3378.045498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3379.305523] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3379.379800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3381.997091] LustreError: 179114:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3381.999851] LustreError: 179114:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3382.058713] Lustre: server umount lustre-OST0000 complete [ 3383.270891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3383.275805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3383.277562] Lustre: Skipped 1 previous similar message [ 3383.775834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3388.384037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3388.385433] Lustre: Skipped 2 previous similar messages [ 3392.483317] LustreError: 179314:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3392.486215] LustreError: 179314:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3392.512925] Lustre: server umount lustre-MDT0000 complete [ 3393.504171] LustreError: 176988:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3393.510673] LustreError: 176988:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3393.822315] LustreError: 176982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408703 with bad export cookie 3431090125501022358 [ 3393.827300] LustreError: 176982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3393.827656] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3404.325229] Lustre: server umount lustre-MDT0001 complete [ 3407.086217] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:45:17 (1754408717) [ 3412.791477] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 3413.805345] Key type lgssc unregistered [ 3413.938439] LNet: 180695:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3413.942963] LNetError: 180695:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3413.952333] LNet: Removed LNI 192.168.204.156@tcp [ 3414.268238] Key type .llcrypt unregistered [ 3414.269721] Key type ._llcrypt unregistered [ 3420.808713] Key type ._llcrypt registered [ 3420.809481] Key type .llcrypt registered [ 3420.839059] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 3421.199955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3421.223681] alg: No test for adler32 (adler32-zlib) [ 3422.076189] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 3422.154666] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 3423.727144] Key type lgssc registered [ 3424.056633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3426.549877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3426.555353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3427.692630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3428.931544] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3431.687889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3431.813897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3433.115164] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3434.873411] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3436.009528] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3438.563185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3438.628656] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3438.629825] LustreError: 183489:0:(llog.c:1642:llog_backup()) MGC192.168.204.156@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3438.632677] Lustre: 183489:0:(mgc_request_server.c:607:mgc_llog_local_copy()) MGC192.168.204.156@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3438.665824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3440.521453] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3442.247207] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3447.265126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3447.265199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3447.267350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3447.272016] Lustre: Skipped 2 previous similar messages [ 3449.019149] LustreError: 184132:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3449.041586] Lustre: server umount lustre-MDT0000 complete [ 3450.158161] LustreError: 183518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408759 with bad export cookie 3253054669661186043 [ 3450.160838] LustreError: 183518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3450.161199] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3450.216084] LustreError: 184337:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3450.217710] LustreError: 184337:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3450.286932] Lustre: server umount lustre-MDT0001 complete [ 3461.744023] LustreError: 184537:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3461.748142] LustreError: 184537:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3461.811680] Lustre: server umount lustre-OST0000 complete [ 3464.597211] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 3467.503668] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 3470.534787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3472.684641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3474.692821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3476.822252] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3479.099239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3479.117022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3479.190147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3479.201436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3479.235636] Lustre: lustre-MDT0000: new disk, initializing [ 3479.257351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3479.263131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3480.573719] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3484.891168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3484.913928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3484.945077] Lustre: 186859:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3484.962671] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3484.963619] Lustre: Skipped 1 previous similar message [ 3484.964827] LustreError: 187617:0:(llog.c:1642:llog_backup()) MGC192.168.204.156@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3484.967101] LustreError: 187617:0:(llog.c:1642:llog_backup()) Skipped 1 previous similar message [ 3484.968798] Lustre: 187617:0:(mgc_request_server.c:630:mgc_llog_local_copy()) MGC192.168.204.156@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3484.977116] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3484.978665] Lustre: Skipped 1 previous similar message [ 3485.012928] Lustre: lustre-MDT0001: new disk, initializing [ 3485.045459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3485.057055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3485.059596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3486.315106] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3489.418093] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3490.572181] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3492.736196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3492.761448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3492.847760] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3492.849556] Lustre: Skipped 1 previous similar message [ 3492.850562] LustreError: 188567:0:(llog.c:1642:llog_backup()) MGC192.168.204.156@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3492.853595] LustreError: 188567:0:(llog.c:1642:llog_backup()) Skipped 1 previous similar message [ 3492.856488] Lustre: 188567:0:(mgc_request_server.c:630:mgc_llog_local_copy()) MGC192.168.204.156@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3492.858850] Lustre: 188567:0:(mgc_request_server.c:630:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3492.866562] Lustre: lustre-OST0000: new disk, initializing [ 3492.868381] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3493.949206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3493.952453] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3493.991521] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3494.781515] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3497.729812] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3499.011549] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3499.084233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3500.350558] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3500.421227] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3504.097811] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3504.098938] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3504.101781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3504.106981] Lustre: Skipped 3 previous similar messages [ 3507.116808] LustreError: 189766:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3507.118573] LustreError: 189766:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3507.179649] Lustre: server umount lustre-OST0000 complete [ 3509.217784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3509.218860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3509.221347] Lustre: Skipped 2 previous similar messages [ 3514.336481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3514.337986] Lustre: Skipped 2 previous similar messages [ 3514.488876] LustreError: 189967:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3514.492095] LustreError: 189967:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3514.519144] Lustre: server umount lustre-MDT0000 complete [ 3515.724066] LustreError: 186850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408825 with bad export cookie 3253054669661186729 [ 3515.726553] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3515.727829] LustreError: 186850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3515.863365] Lustre: server umount lustre-MDT0001 complete [ 3518.607648] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:47:08 (1754408828) [ 3523.515588] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 3526.745884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3526.869077] LustreError: 191528:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3526.876207] LustreError: 191528:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3526.884123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3526.885898] Lustre: Skipped 1 previous similar message [ 3528.094718] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3530.859408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3532.291619] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3533.187708] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3535.419073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3537.555208] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3540.411446] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3540.436745] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3540.498991] Lustre: lustre-OST0001: new disk, initializing [ 3540.501600] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3542.370034] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3542.488600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3542.493798] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3542.534676] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3547.824664] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3551.533929] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3557.259226] Lustre: DEBUG MARKER: ADD OST3 [ 3558.977995] loop: module loaded [ 3558.997072] loop0: detected capacity change from 0 to 204800000 [ 3559.053585] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3559.056321] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3559.059359] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3559.112212] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3561.074201] loop0: detected capacity change from 0 to 204800000 [ 3563.012286] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3563.034368] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3563.102799] Lustre: lustre-OST0003: new disk, initializing [ 3563.104560] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3563.141990] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3563.144644] Lustre: Skipped 3 previous similar messages [ 3564.063141] Lustre: DEBUG MARKER: STOP OST3 [ 3564.512089] Lustre: *** cfs_fail_loc=2109, val=2*** [ 3564.557262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3564.560645] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3564.574498] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 3564.760602] Lustre: Failing over lustre-OST0003 [ 3564.785337] LustreError: 195874:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 3564.787722] LustreError: 195874:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3564.813378] Lustre: server umount lustre-OST0003 complete [ 3565.280402] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3566.559119] LustreError: 195788:0:(osp_precreate.c:1283:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 3569.124560] Lustre: DEBUG MARKER: START OST3 again [ 3569.640398] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3569.642299] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3569.644602] LustreError: 192968:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3569.650424] LustreError: 192968:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3571.218378] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3571.303503] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 3572.910567] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3573.222925] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 192.168.204.156@tcp (at 0@lo) [ 3573.223844] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3573.282573] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3587.109564] Lustre: Failing over lustre-OST0003 [ 3587.136491] LustreError: 197783:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 3587.139683] LustreError: 197783:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3587.158214] Lustre: server umount lustre-OST0003 complete [ 3588.577437] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3588.580251] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3588.584711] Lustre: Skipped 1 previous similar message [ 3588.586964] LustreError: 197036:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3588.592977] LustreError: 197036:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3592.166466] LustreError: 192968:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3592.172607] LustreError: 192968:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3597.290308] LustreError: 193941:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3597.296076] LustreError: 193941:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3602.402791] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3602.404194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3602.406529] Lustre: Skipped 4 previous similar messages [ 3602.408670] Lustre: Skipped 2 previous similar messages [ 3605.854303] Lustre: server umount lustre-MDT0000 complete [ 3607.195527] LustreError: 191508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408916 with bad export cookie 3253054669661187597 [ 3607.200170] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3607.200182] LustreError: 191508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3607.520696] LustreError: 197526:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3607.527439] LustreError: 197526:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3612.640401] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3612.642408] Lustre: Skipped 3 previous similar messages [ 3613.514135] Lustre: server umount lustre-MDT0001 complete [ 3614.856506] Lustre: server umount lustre-OST0000 complete [ 3616.180671] Lustre: server umount lustre-OST0001 complete [ 3618.224721] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 3620.720459] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 3624.547220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3626.921157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3628.929674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3631.040420] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3633.175632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3633.198466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3633.281766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3633.294996] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3633.331094] Lustre: lustre-MDT0000: new disk, initializing [ 3633.355637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3633.357312] Lustre: Skipped 1 previous similar message [ 3633.362575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3634.656720] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3638.884430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3638.911678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3638.943170] Lustre: 200823:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3639.048119] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3640.413332] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3643.601529] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3644.908018] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3647.427534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3647.456266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3648.954409] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3648.990803] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3649.501296] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3652.696404] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3654.155375] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3654.233717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3655.715229] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3655.796829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3658.541108] LustreError: 203733:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3658.544016] LustreError: 203733:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 3658.602894] Lustre: server umount lustre-OST0000 complete [ 3664.352315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3664.353154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3664.355026] Lustre: Skipped 3 previous similar messages [ 3674.079092] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3674.135841] Lustre: server umount lustre-MDT0000 complete [ 3674.592101] Lustre: 181176:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754408968/real 1754408968] req@ffff9f8a467b4380 x1839630889700864/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1754408984 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 3674.592633] LustreError: 201876:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3674.608360] LustreError: 201876:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3675.409131] LustreError: 201598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408985 with bad export cookie 3253054669661189851 [ 3675.412844] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3675.544159] Lustre: server umount lustre-MDT0001 complete [ 3676.743175] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:49:46 (1754408986) [ 3681.808112] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 3685.309741] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 3688.555038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3690.668867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3692.756921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3694.894411] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3697.097292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3697.123032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3697.197120] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3697.207449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3697.209068] Lustre: Skipped 3 previous similar messages [ 3697.240878] Lustre: lustre-MDT0000: new disk, initializing [ 3697.242040] Lustre: Skipped 2 previous similar messages [ 3697.267349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3697.269601] Lustre: Skipped 2 previous similar messages [ 3698.513311] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3702.713668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3702.736889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3702.777072] Lustre: 206963:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3702.878188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3702.880246] Lustre: Skipped 3 previous similar messages [ 3702.891183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3704.219755] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3707.320268] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3708.600241] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3711.051140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3711.077549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3712.673342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3713.308724] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3716.676890] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3718.197672] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3718.280432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3719.718401] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3719.799715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3722.536342] Lustre: server umount lustre-OST0000 complete [ 3728.352660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3728.352721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3728.353278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3728.353283] Lustre: Skipped 3 previous similar messages [ 3728.356028] Lustre: Skipped 2 previous similar messages [ 3728.357713] LustreError: Skipped 1 previous similar message [ 3738.079165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3738.111653] LustreError: 210069:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3738.113106] LustreError: 210069:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3738.138145] Lustre: server umount lustre-MDT0000 complete [ 3738.591260] Lustre: 181175:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754409032/real 1754409032] req@ffff9f8b46a21180 x1839630889732352/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1754409048 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 3738.592143] LustreError: 207752:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3738.599518] Lustre: 181175:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3738.608063] LustreError: 207752:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3739.338304] LustreError: 207737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754409048 with bad export cookie 3253054669661190733 [ 3739.341824] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3739.342497] LustreError: 207737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3739.472801] Lustre: server umount lustre-MDT0001 complete [ 3742.150532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3743.611744] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3746.255141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3747.561795] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3749.300679] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3750.422264] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3752.635903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3754.645410] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3756.501715] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3778.532974] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3784.381117] Lustre: server umount lustre-OST0000 complete [ 3793.888099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3793.889894] Lustre: Skipped 11 previous similar messages [ 3800.031121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3800.090624] Lustre: server umount lustre-MDT0000 complete [ 3801.420874] LustreError: 210773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754409111 with bad export cookie 3253054669661191615 [ 3801.424519] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3801.425361] LustreError: 210773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3801.576858] Lustre: server umount lustre-MDT0001 complete [ 3890.948949] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 3892.067674] Key type lgssc unregistered [ 3892.211367] LNet: 214055:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3892.213950] LNetError: 214055:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3892.221435] LNet: Removed LNI 192.168.204.156@tcp [ 3892.555167] Key type .llcrypt unregistered [ 3892.556197] Key type ._llcrypt unregistered [ 3901.110757] Key type ._llcrypt registered [ 3901.111760] Key type .llcrypt registered [ 3901.151986] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 3907.778217] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 3908.120822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3908.209456] alg: No test for adler32 (adler32-zlib) [ 3909.061226] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 3909.152464] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 3910.735161] Key type lgssc registered [ 3911.118918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3914.329925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3916.793662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3918.952336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3921.180520] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3923.549427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3923.570843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3923.578257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3924.666430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3924.678500] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3924.722045] Lustre: lustre-MDT0000: new disk, initializing [ 3924.747391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3924.754101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3926.122535] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3930.476770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3930.499513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3930.530925] Lustre: 217343:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3930.555581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3930.557137] Lustre: Skipped 1 previous similar message [ 3930.593680] Lustre: lustre-MDT0001: new disk, initializing [ 3930.632355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3930.646505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3930.651683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3932.019697] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3935.149619] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3936.388095] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3938.695921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3938.725571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3938.840874] Lustre: lustre-OST0000: new disk, initializing [ 3938.842607] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3938.873761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3939.773056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3939.776834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3939.829336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3940.809882] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3943.837193] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3945.249625] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3945.325675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3946.644512] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3946.714301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3950.049489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3950.050642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3950.052267] LustreError: Skipped 1 previous similar message [ 3950.052756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3950.058709] Lustre: Skipped 1 previous similar message [ 3950.063897] Lustre: Skipped 1 previous similar message [ 3953.453155] LustreError: 220252:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3953.527514] Lustre: server umount lustre-OST0000 complete [ 3955.169733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3955.170742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3955.173653] Lustre: Skipped 1 previous similar message [ 3960.288794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3960.290467] Lustre: Skipped 2 previous similar messages [ 3960.892410] LustreError: 220453:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3960.894321] LustreError: 220453:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3960.920960] Lustre: server umount lustre-MDT0000 complete [ 3962.200391] LustreError: 219076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754409271 with bad export cookie 11205495403942651359 [ 3962.203318] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3962.206066] LustreError: 219076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3962.273153] LustreError: 220656:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3962.275380] LustreError: 220656:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3962.364563] Lustre: server umount lustre-MDT0001 complete [ 3968.544956] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 3971.966398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3972.127551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3973.622484] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3976.465714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3976.582732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3977.884957] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3978.821326] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3981.021661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3983.036939] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3985.875977] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3985.895798] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3985.950492] Lustre: lustre-OST0001: new disk, initializing [ 3985.952178] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3985.983853] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3985.985914] Lustre: Skipped 1 previous similar message [ 3987.891197] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3993.093399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3993.097659] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3993.127755] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3993.323587] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3999.035911] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4007.880685] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:55:18 (1754409318) [ 4012.511898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4012.513961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4012.517687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4014.715885] LustreError: 225175:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4014.717658] LustreError: 225175:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4014.745611] Lustre: server umount lustre-MDT0000 complete [ 4015.984258] LustreError: 222913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754409325 with bad export cookie 11205495403942652227 [ 4015.987984] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4015.990327] LustreError: 222913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4016.131541] Lustre: server umount lustre-MDT0001 complete [ 4027.892134] LustreError: 225576:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4027.895756] LustreError: 225576:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4027.912909] Lustre: server umount lustre-OST0000 complete [ 4039.410102] LustreError: 225779:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4039.412649] LustreError: 225779:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4039.482150] Lustre: server umount lustre-OST0001 complete [ 4041.494573] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 4044.744196] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 4048.106612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4050.368213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4052.423508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4054.581665] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4056.935370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4056.954191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4057.034785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4057.045568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4057.081779] Lustre: lustre-MDT0000: new disk, initializing [ 4057.103914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4057.109935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4058.459524] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4062.586126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4062.609081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4062.642173] Lustre: 227950:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4062.742061] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4064.078775] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4067.220544] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4068.478778] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4070.778463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4070.804159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4070.898410] Lustre: lustre-OST0000: new disk, initializing [ 4070.899540] Lustre: Skipped 1 previous similar message [ 4070.900989] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4070.902975] Lustre: Skipped 2 previous similar messages [ 4072.375772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4072.379038] Lustre: Skipped 1 previous similar message [ 4072.382377] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4072.412849] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4072.786473] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4075.953805] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4077.324398] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4077.398660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4078.816241] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4078.902596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4082.657399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4082.658032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4082.660898] LustreError: Skipped 1 previous similar message [ 4082.664205] Lustre: Skipped 3 previous similar messages [ 4082.667057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4082.669569] Lustre: Skipped 5 previous similar messages [ 4085.621113] LustreError: 230856:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4085.624108] LustreError: 230856:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4085.685461] Lustre: server umount lustre-OST0000 complete [ 4087.777230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4087.780802] Lustre: Skipped 1 previous similar message [ 4092.897266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4092.899133] Lustre: Skipped 3 previous similar messages [ 4093.081497] Lustre: server umount lustre-MDT0000 complete [ 4094.405621] LustreError: 227941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754409404 with bad export cookie 11205495403942654355 [ 4094.409140] LustreError: 227941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4094.409473] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4094.577616] Lustre: server umount lustre-MDT0001 complete [ 4097.486362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4097.643612] LustreError: 231733:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4097.650474] LustreError: 231733:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4097.674758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4097.676880] Lustre: Skipped 2 previous similar messages [ 4099.037225] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4101.999675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4103.513800] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4105.381690] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4106.620159] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4108.954819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4111.032659] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4112.860801] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4260.835438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4260.835895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4260.836388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4260.839404] Lustre: Skipped 1 previous similar message [ 4260.844624] LustreError: Skipped 1 previous similar message [ 4263.211116] LustreError: 233998:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4263.213461] LustreError: 233998:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4263.273848] Lustre: server umount lustre-OST0000 complete [ 4265.955613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4270.755234] Lustre: server umount lustre-MDT0000 complete [ 4271.072490] LustreError: 231733:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4271.078967] LustreError: 231733:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4272.122105] LustreError: 231715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754409581 with bad export cookie 11205495403942655237 [ 4272.124980] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4272.126357] LustreError: 231715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4272.271466] Lustre: server umount lustre-MDT0001 complete [ 4377.601642] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 4378.677189] Key type lgssc unregistered [ 4378.816376] LNet: 234974:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4378.819690] LNetError: 234974:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4378.828360] LNet: Removed LNI 192.168.204.156@tcp [ 4379.150222] Key type .llcrypt unregistered [ 4379.151389] Key type ._llcrypt unregistered [ 4387.508054] Key type ._llcrypt registered [ 4387.509171] Key type .llcrypt registered [ 4387.546270] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 4393.681307] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 4393.984828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4393.994414] alg: No test for adler32 (adler32-zlib) [ 4394.861895] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 4394.963733] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 4396.551155] Key type lgssc registered [ 4396.920753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4400.008557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4402.277561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4404.352381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4406.598035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4411.840478] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 4415.204971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4415.223615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4415.229027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4416.312813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4416.327128] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4416.362771] Lustre: lustre-MDT0000: new disk, initializing [ 4416.389066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4416.395854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4417.847743] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4422.155689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4422.177148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4422.212526] Lustre: 238955:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4422.234468] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4422.237259] Lustre: Skipped 1 previous similar message [ 4422.273274] Lustre: lustre-MDT0001: new disk, initializing [ 4422.307640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4422.321299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4422.324947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4423.655483] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4425.900783] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4428.077072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4428.100107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4428.191744] Lustre: lustre-OST0000: new disk, initializing [ 4428.193960] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4428.227437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4430.121616] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4430.455214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4430.458448] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4430.508412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4434.224254] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4434.249050] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4434.304849] Lustre: lustre-OST0001: new disk, initializing [ 4434.306983] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4434.339932] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4435.782069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4435.785121] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4435.828311] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4436.265909] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4441.788802] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4446.479796] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4456.422658] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4456.424744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4456.426303] Lustre: Skipped 2 previous similar messages [ 4456.428018] Lustre: Skipped 1 previous similar message [ 4460.421346] LustreError: 242454:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4460.453255] Lustre: server umount lustre-MDT0000 complete [ 4461.722137] LustreError: 238946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754409771 with bad export cookie 1642201121260560021 [ 4461.724486] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4461.727721] LustreError: 238946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4461.786289] LustreError: 242655:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4461.788156] LustreError: 242655:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4461.863873] Lustre: server umount lustre-MDT0001 complete [ 4472.819059] LustreError: 242855:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4472.822394] LustreError: 242855:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4472.841845] Lustre: server umount lustre-OST0000 complete [ 4484.274522] LustreError: 243057:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4484.277700] LustreError: 243057:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4484.348199] Lustre: server umount lustre-OST0001 complete [ 4488.947809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4491.857652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4492.215190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4494.983086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4497.763568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4498.112317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4504.207547] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 4507.714290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4507.719654] Lustre: lustre-MDT0000: reset Object Index mappings [ 4507.866340] LustreError: 246358:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4507.873301] LustreError: 246358:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 4507.885765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4509.249675] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4512.141990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4512.147296] Lustre: lustre-MDT0001: reset Object Index mappings [ 4513.623350] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4514.587972] Lustre: 247456:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4516.792082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4516.906410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4516.908118] Lustre: Skipped 1 previous similar message [ 4517.922322] LustreError: 247814:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4517.926047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4518.715771] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4521.532327] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4522.666975] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4523.601829] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4526.816613] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4528.005750] Lustre: 249288:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4537.823907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4537.826276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4537.830383] Lustre: Skipped 1 previous similar message [ 4537.832125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4537.833828] Lustre: Skipped 1 previous similar message [ 4540.219982] LustreError: 249392:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4540.221579] LustreError: 249392:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4540.252379] Lustre: server umount lustre-MDT0000 complete [ 4541.490212] LustreError: 246338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754409851 with bad export cookie 1642201121260566797 [ 4541.493597] LustreError: 246338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4541.493947] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4541.640603] Lustre: server umount lustre-MDT0001 complete [ 4552.628578] LustreError: 249794:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4552.630899] LustreError: 249794:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4552.648349] Lustre: server umount lustre-OST0000 complete [ 4564.145921] Lustre: server umount lustre-OST0001 complete [ 4570.675422] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 12:04:40 (1754409880) [ 4576.131173] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 4579.307394] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 4582.708200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4584.806378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4586.711164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4588.749519] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4590.985584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4591.003193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4591.081925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4591.093266] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4591.129821] Lustre: lustre-MDT0000: new disk, initializing [ 4591.152425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4591.154019] Lustre: Skipped 1 previous similar message [ 4591.159315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4592.474360] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4596.574607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4596.595618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4596.627333] Lustre: 253499:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4596.721040] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4598.014655] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4601.035099] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4602.222939] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4604.534908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4604.564053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4604.666819] Lustre: lustre-OST0000: new disk, initializing [ 4604.668424] Lustre: Skipped 1 previous similar message [ 4604.670561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4604.673522] Lustre: Skipped 2 previous similar messages [ 4606.396360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4606.399034] Lustre: Skipped 1 previous similar message [ 4606.401420] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4606.441928] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4606.750886] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4609.851630] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4611.204807] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4611.274799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4612.552133] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4612.620037] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4616.673368] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4616.673478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4616.676153] LustreError: Skipped 1 previous similar message [ 4616.679339] Lustre: Skipped 3 previous similar messages [ 4616.680137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4616.684583] Lustre: Skipped 5 previous similar messages [ 4619.374115] LustreError: 256407:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4619.375779] LustreError: 256407:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4619.435451] Lustre: server umount lustre-OST0000 complete [ 4621.792443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4621.793303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4621.796407] Lustre: Skipped 2 previous similar messages [ 4626.777246] Lustre: server umount lustre-MDT0000 complete [ 4626.912031] LustreError: 253510:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4626.916940] LustreError: 253510:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4628.043564] LustreError: 253491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754409937 with bad export cookie 1642201121260568883 [ 4628.048237] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4628.191357] Lustre: server umount lustre-MDT0001 complete [ 4634.431216] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 4638.001114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4638.147401] LustreError: 257977:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4638.166608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4638.168792] Lustre: Skipped 2 previous similar messages [ 4639.530535] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4642.455365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4644.007681] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4645.054827] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4647.474171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4649.687690] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4652.707118] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4652.731893] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4652.793315] Lustre: lustre-OST0001: new disk, initializing [ 4652.795477] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4653.955619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4653.958945] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4653.988511] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4654.733495] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4659.304737] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4660.617708] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4667.386518] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4667.391857] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4669.412480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4669.413747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4669.416392] Lustre: Skipped 2 previous similar messages [ 4669.420633] Lustre: Skipped 4 previous similar messages [ 4675.323684] LustreError: 261457:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4675.325892] LustreError: 261457:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4675.353127] Lustre: server umount lustre-MDT0000 complete [ 4676.506340] LustreError: 258689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754409986 with bad export cookie 1642201121260569772 [ 4676.510400] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4676.510550] LustreError: 258689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4676.644162] Lustre: server umount lustre-MDT0001 complete [ 4687.873263] Lustre: server umount lustre-OST0000 complete [ 4699.319816] Lustre: server umount lustre-OST0001 complete [ 4704.260318] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 4707.449726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4707.579247] LustreError: 263231:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4707.596285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4707.598085] Lustre: Skipped 3 previous similar messages [ 4708.834888] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4711.602188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4713.067323] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4714.180510] Lustre: 264331:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4716.534631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4717.666739] LustreError: 264687:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4717.674330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4718.494865] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4721.199952] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4722.283504] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4723.154064] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4726.293439] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4727.428748] Lustre: 266167:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4734.121601] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4734.125795] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4736.900460] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 12:07:27 (1754410047) [ 4738.018417] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4738.020761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4738.021428] Lustre: Skipped 3 previous similar messages [ 4738.024967] Lustre: Skipped 7 previous similar messages [ 4743.930779] LustreError: 266854:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4743.932950] LustreError: 266854:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4743.957786] Lustre: server umount lustre-MDT0000 complete [ 4745.148901] LustreError: 263210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754410054 with bad export cookie 1642201121260576044 [ 4745.151848] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4745.153380] LustreError: 263210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4745.282450] Lustre: server umount lustre-MDT0001 complete [ 4756.486146] Lustre: server umount lustre-OST0000 complete [ 4768.043471] Lustre: server umount lustre-OST0001 complete [ 4773.395538] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 4776.637147] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 4780.070452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4782.412222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4784.613347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4786.851106] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4789.536083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4789.557319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4789.659402] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4789.670450] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4789.711819] Lustre: lustre-MDT0000: new disk, initializing [ 4789.743947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4791.150842] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4795.496806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4795.521267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4795.560255] Lustre: 270286:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4795.673578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4797.113599] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4800.352453] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4801.606359] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4803.846893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4803.870528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4805.669395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4805.767786] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 4808.776621] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4809.428550] Lustre: Mounted lustre-client [ 4809.724430] LustreError: 272770:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff9f8a436de000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 4809.750844] Lustre: Unmounted lustre-client [ 4814.818350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4814.821944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4814.823548] Lustre: Skipped 2 previous similar messages [ 4814.825505] Lustre: Skipped 4 previous similar messages [ 4818.721536] Lustre: server umount lustre-OST0000 complete [ 4821.471677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4834.271122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4834.324231] Lustre: server umount lustre-MDT0000 complete [ 4835.295987] LustreError: 270296:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4835.300509] LustreError: 270296:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4835.559074] LustreError: 270278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754410145 with bad export cookie 1642201121260583450 [ 4835.561688] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4835.561980] LustreError: 270278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4835.701694] Lustre: server umount lustre-MDT0001 complete [ 4839.136162] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 4840.224826] Key type lgssc unregistered [ 4871.071327] LNet: 274043:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4871.073733] LNetError: 274043:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4941.732380] LNet: Removed LNI 192.168.204.156@tcp [ 4942.055097] Key type .llcrypt unregistered [ 4942.056155] Key type ._llcrypt unregistered [ 4947.943588] Key type ._llcrypt registered [ 4947.944380] Key type .llcrypt registered [ 4947.981738] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 4957.843853] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 4958.901557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4958.925981] alg: No test for adler32 (adler32-zlib) [ 4960.061297] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 4960.276556] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 4961.991197] Key type lgssc registered [ 4962.958460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4969.420720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4975.431221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4984.331231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4993.749147] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4997.017233] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 12:11:44 (1754410304) [ 5057.373636] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 12:12:46 (1754410366) [ 5086.035507] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 5088.233396] Key type lgssc unregistered [ 5088.480435] LNet: 279775:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5088.485688] LNetError: 279775:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5088.498435] LNet: Removed LNI 192.168.204.156@tcp [ 5089.011839] Key type .llcrypt unregistered [ 5089.013460] Key type ._llcrypt unregistered [ 5146.083700] Key type ._llcrypt registered [ 5146.085404] Key type .llcrypt registered [ 5146.184665] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 5146.444379] Key type .llcrypt unregistered [ 5146.451625] Key type ._llcrypt unregistered [ 5167.034090] Key type ._llcrypt registered [ 5167.035724] Key type .llcrypt registered [ 5167.121977] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 5167.316285] Key type .llcrypt unregistered [ 5167.317844] Key type ._llcrypt unregistered [ 5217.808995] Key type ._llcrypt registered [ 5217.810500] Key type .llcrypt registered [ 5217.897937] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 5218.120604] Key type .llcrypt unregistered [ 5218.122616] Key type ._llcrypt unregistered [ 5239.768870] Key type ._llcrypt registered [ 5239.770176] Key type .llcrypt registered [ 5239.832597] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 5240.058591] Key type .llcrypt unregistered [ 5240.060122] Key type ._llcrypt unregistered [ 5261.019205] Key type ._llcrypt registered [ 5261.029373] Key type .llcrypt registered [ 5261.107933] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 5261.321456] Key type .llcrypt unregistered [ 5261.323080] Key type ._llcrypt unregistered [ 5322.764365] Key type ._llcrypt registered [ 5322.766043] Key type .llcrypt registered [ 5322.876965] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 5323.154288] Key type .llcrypt unregistered [ 5323.156071] Key type ._llcrypt unregistered [ 5345.498958] Key type ._llcrypt registered [ 5345.502405] Key type .llcrypt registered [ 5345.569622] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 5345.751748] Key type .llcrypt unregistered [ 5345.753202] Key type ._llcrypt unregistered [ 5362.498602] Key type ._llcrypt registered [ 5362.500252] Key type .llcrypt registered [ 5362.552571] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 5362.689557] Key type .llcrypt unregistered [ 5362.691415] Key type ._llcrypt unregistered [ 5390.750655] Key type ._llcrypt registered [ 5390.752674] Key type .llcrypt registered [ 5390.807205] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 5398.531237] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 5399.001228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5399.012635] alg: No test for adler32 (adler32-zlib) [ 5399.877159] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 5399.971827] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 5401.559157] Key type lgssc registered [ 5401.932103] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5405.712583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5408.784183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5410.968685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5413.406053] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5414.050675] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5252 sec ========== 12:18:44 (1754410724) [ 5414.683167] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:18:44 (1754410724) === [ 5416.031530] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:18:46 (1754410726) === [ 5430.584205] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 5431.779598] Key type lgssc unregistered [ 5431.931572] LNet: 295613:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5431.934740] LNetError: 295613:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5431.942762] LNet: Removed LNI 192.168.204.156@tcp [ 5432.259054] Key type .llcrypt unregistered [ 5432.260710] Key type ._llcrypt unregistered