[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 474812754 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002245] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.006958] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.010043] LSM: Security Framework initializing [ 0.011049] Yama: becoming mindful. [ 0.012036] SELinux: Initializing. [ 0.013069] *** VALIDATE selinux *** [ 0.022263] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026576] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028043] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029100] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030109] *** VALIDATE tmpfs *** [ 0.031433] *** VALIDATE proc *** [ 0.032238] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.035217] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037024] Spectre V2 : User space: Vulnerable [ 0.038007] Speculative Store Bypass: Vulnerable [ 0.041984] debug: unmapping init [mem 0xffffffff8de59000-0xffffffff8de60fff] [ 0.043171] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044717] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045025] ... version: 2 [ 0.046013] ... bit width: 48 [ 0.047014] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049013] ... max period: 00007fffffffffff [ 0.050016] ... fixed-purpose events: 3 [ 0.051016] ... event mask: 000000070000000f [ 0.052301] rcu: Hierarchical SRCU implementation. [ 0.054477] smp: Bringing up secondary CPUs ... [ 0.055631] x86: Booting SMP configuration: [ 0.056025] .... node #0, CPUs: #1 #2 #3 [ 0.061096] smp: Brought up 1 node, 4 CPUs [ 0.063021] smpboot: Max logical packages: 1 [ 0.064013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145896] node 0 deferred pages initialised in 79ms [ 0.147307] devtmpfs: initialized [ 0.149269] x86/mm: Memory block size: 128MB [ 0.152674] gcov: version magic: 0x41383552 [ 0.155146] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157270] pinctrl core: initialized pinctrl subsystem [ 0.158217] [ 0.158776] ************************************************************* [ 0.159015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160015] ** ** [ 0.161012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162018] ** ** [ 0.163013] ** This means that this kernel is built to expose internal ** [ 0.164016] ** IOMMU data structures, which may compromise security on ** [ 0.165015] ** your system. ** [ 0.166013] ** ** [ 0.167017] ** If you see this message and you are not debugging the ** [ 0.168014] ** kernel, report this immediately to your vendor! ** [ 0.169017] ** ** [ 0.170010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171011] ************************************************************* [ 0.172845] NET: Registered protocol family 16 [ 0.173437] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.174067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.175070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.177123] cpuidle: using governor menu [ 0.178865] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.181476] PCI: Using configuration type 1 for base access [ 0.184123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.192134] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.195084] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199262] cryptd: max_cpu_qlen set to 1000 [ 0.202028] ACPI: Added _OSI(Module Device) [ 0.204013] ACPI: Added _OSI(Processor Device) [ 0.205011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.206009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.210132] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.216012] ACPI: Interpreter enabled [ 0.217069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.219013] ACPI: Using IOAPIC for interrupt routing [ 0.221100] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.224472] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.234909] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.237044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.239018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.242083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.247460] acpiphp: Slot [2] registered [ 0.249134] acpiphp: Slot [5] registered [ 0.251166] acpiphp: Slot [6] registered [ 0.252108] acpiphp: Slot [7] registered [ 0.253128] acpiphp: Slot [8] registered [ 0.255132] acpiphp: Slot [9] registered [ 0.256171] acpiphp: Slot [10] registered [ 0.258402] acpiphp: Slot [3] registered [ 0.260231] acpiphp: Slot [4] registered [ 0.262122] acpiphp: Slot [11] registered [ 0.264257] acpiphp: Slot [12] registered [ 0.266134] acpiphp: Slot [13] registered [ 0.268154] acpiphp: Slot [14] registered [ 0.270131] acpiphp: Slot [15] registered [ 0.272116] acpiphp: Slot [16] registered [ 0.274124] acpiphp: Slot [17] registered [ 0.276107] acpiphp: Slot [18] registered [ 0.278136] acpiphp: Slot [19] registered [ 0.280151] acpiphp: Slot [20] registered [ 0.282141] acpiphp: Slot [21] registered [ 0.284113] acpiphp: Slot [22] registered [ 0.286118] acpiphp: Slot [23] registered [ 0.287121] acpiphp: Slot [24] registered [ 0.289105] acpiphp: Slot [25] registered [ 0.290101] acpiphp: Slot [26] registered [ 0.292151] acpiphp: Slot [27] registered [ 0.294121] acpiphp: Slot [28] registered [ 0.295082] acpiphp: Slot [29] registered [ 0.297115] acpiphp: Slot [30] registered [ 0.298088] acpiphp: Slot [31] registered [ 0.299049] PCI host bridge to bus 0000:00 [ 0.301020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.303024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.306023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.308032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.311025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.315028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.316280] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.319926] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.322160] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.332022] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.338057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.341026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.342014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.344019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.348293] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.351972] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.354047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.357199] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.363018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.376016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.381013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.387013] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.397017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.403016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.412015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.422392] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.427016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.437015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.453021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.464754] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.474020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.484020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.512023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.523528] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.533036] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.541019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.561018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.571396] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.581020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.599038] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.636022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.649312] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.662019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.672019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.693017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.705429] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.707390] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.709344] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.711459] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.713215] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.717208] iommu: Default domain type: Passthrough [ 0.719682] SCSI subsystem initialized [ 0.721182] ACPI: bus type USB registered [ 0.723134] usbcore: registered new interface driver usbfs [ 0.725104] usbcore: registered new interface driver hub [ 0.726074] usbcore: registered new device driver usb [ 0.728211] pps_core: LinuxPPS API ver. 1 registered [ 0.729000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.731061] PTP clock support registered [ 0.733130] EDAC MC: Ver: 3.0.0 [ 0.735542] PCI: Using ACPI for IRQ routing [ 0.738235] NetLabel: Initializing [ 0.739010] NetLabel: domain hash size = 128 [ 0.741010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.743084] NetLabel: unlabeled traffic allowed by default [ 0.746109] vgaarb: loaded [ 0.747251] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.749014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.754000] clocksource: Switched to clocksource kvm-clock [ 0.865796] VFS: Disk quotas dquot_6.6.0 [ 0.866697] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.868086] *** VALIDATE ramfs *** [ 0.869177] *** VALIDATE hugetlbfs *** [ 0.870715] pnp: PnP ACPI init [ 0.872856] pnp: PnP ACPI: found 6 devices [ 0.888982] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.892574] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.894577] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.896370] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.898443] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.900511] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.903491] NET: Registered protocol family 2 [ 0.906320] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.912170] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.916242] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.922166] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.926100] TCP: Hash tables configured (established 65536 bind 65536) [ 0.929116] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.932445] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.935420] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.938519] NET: Registered protocol family 1 [ 0.942116] RPC: Registered named UNIX socket transport module. [ 0.944214] RPC: Registered udp transport module. [ 0.945938] RPC: Registered tcp transport module. [ 0.947625] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.949896] NET: Registered protocol family 44 [ 0.951586] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.954357] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.957041] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.959558] PCI: CLS 0 bytes, default 64 [ 0.960819] Unpacking initramfs... [ 2.400163] debug: unmapping init [mem 0xffff8aa9bcc54000-0xffff8aa9bffbffff] [ 2.403915] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.406174] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.409105] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.899222] Initialise system trusted keyrings [ 2.900633] Key type blacklist registered [ 2.902541] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.910600] zbud: loaded [ 2.913702] *** VALIDATE nfs *** [ 2.914996] *** VALIDATE nfs4 *** [ 2.916606] pstore: using deflate compression [ 2.919827] Platform Keyring initialized [ 3.029377] NET: Registered protocol family 38 [ 3.031305] Key type asymmetric registered [ 3.032781] Asymmetric key parser 'x509' registered [ 3.034816] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.038038] io scheduler mq-deadline registered [ 3.042888] io scheduler kyber registered [ 3.044673] io scheduler bfq registered [ 3.046540] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.049874] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.052529] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.054846] ACPI: Power Button [PWRF] [ 3.060166] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.066701] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.081332] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.089777] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.099089] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.124195] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.150468] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.153779] Non-volatile memory driver v1.3 [ 3.154765] Linux agpgart interface v0.103 [ 3.180599] virtio_blk virtio1: [vda] 150008 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.183497] vda: detected capacity change from 0 to 76804096 [ 3.197081] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.200053] vdb: detected capacity change from 0 to 1073741824 [ 3.214054] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.215887] vdc: detected capacity change from 0 to 2621440000 [ 3.226379] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.228544] vdd: detected capacity change from 0 to 2621440000 [ 3.242880] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.245337] vde: detected capacity change from 0 to 4294967296 [ 3.260491] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.263601] vdf: detected capacity change from 0 to 4294967296 [ 3.270952] libphy: Fixed MDIO Bus: probed [ 3.277490] usbcore: registered new interface driver usbserial_generic [ 3.279901] usbserial: USB Serial support registered for generic [ 3.281595] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.286409] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.288560] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.291392] mousedev: PS/2 mouse device common for all mice [ 3.294281] rtc_cmos 00:05: RTC can wake from S4 [ 3.296783] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.301067] rtc_cmos 00:05: registered as rtc0 [ 3.303228] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.304236] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.311083] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.311475] intel_pstate: CPU model not supported [ 3.318344] hid: raw HID events driver (C) Jiri Kosina [ 3.320472] usbcore: registered new interface driver usbhid [ 3.322504] usbhid: USB HID core driver [ 3.324219] drop_monitor: Initializing network drop monitor service [ 3.326407] Initializing XFRM netlink socket [ 3.328169] NET: Registered protocol family 10 [ 3.330534] Segment Routing with IPv6 [ 3.331510] NET: Registered protocol family 17 [ 3.334075] mpls_gso: MPLS GSO support [ 3.340935] RAS: Correctable Errors collector initialized. [ 3.343043] AVX version of gcm_enc/dec engaged. [ 3.344798] AES CTR mode by8 optimization enabled [ 3.428825] sched_clock: Marking stable (3428772424, 0)->(4320543234, -891770810) [ 3.435594] registered taskstats version 1 [ 3.438980] Loading compiled-in X.509 certificates [ 3.441408] zswap: loaded using pool lzo/zbud [ 3.465493] Key type big_key registered [ 3.479088] Key type encrypted registered [ 3.480393] ima: No TPM chip found, activating TPM-bypass! [ 3.481902] ima: Allocated hash algorithm: sha1 [ 3.483457] ima: No architecture policies found [ 3.484797] evm: Initialising EVM extended attributes: [ 3.486490] evm: security.selinux [ 3.487512] evm: security.ima [ 3.488357] evm: security.capability [ 3.489330] evm: HMAC attrs: 0x1 [ 3.491652] rtc_cmos 00:05: setting system clock to 2026-09-15 12:34:32 UTC (1789475672) [ 3.497737] debug: unmapping init [mem 0xffffffff8ee03000-0xffffffff8effffff] [ 3.500684] debug: unmapping init [mem 0xffffffff8db82000-0xffffffff8de58fff] [ 3.509104] Write protecting the kernel read-only data: 28672k [ 3.512533] debug: unmapping init [mem 0xffffffff8c203000-0xffffffff8c3fffff] [ 3.515051] debug: unmapping init [mem 0xffffffff8cb14000-0xffffffff8cbfffff] [ 3.553282] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.561285] systemd[1]: Detected virtualization kvm. [ 3.564465] systemd[1]: Detected architecture x86-64. [ 3.568807] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.596782] systemd[1]: No hostname configured. [ 3.598228] systemd[1]: Set hostname to . [ 3.599849] random: systemd: uninitialized urandom read (16 bytes read) [ 3.601957] systemd[1]: Initializing machine ID from random generator. [ 3.755218] random: systemd: uninitialized urandom read (16 bytes read) [ 3.757333] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.760717] random: systemd: uninitialized urandom read (16 bytes read) [ 3.762960] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.767219] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.681561] device-mapper: uevent: version 1.0.3 [ 4.684191] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ 4.976866] random: fast init done [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.450986] virtio_net virtio0 ens2: renamed from eth0 [ 5.465617] scsi host0: ata_piix [ 5.472219] scsi host1: ata_piix [ 5.473863] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.532706] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.997387] random: crng init done [ 10.002194] random: 7 urandom warning(s) missed due to ratelimiting [ 11.604822] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 13.027121] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.952824] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.871440] SELinux: Disabled at runtime. [ 17.022665] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.049241] systemd[1]: Detected virtualization kvm. [ 17.051479] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.560495] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.564363] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.587231] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.600555] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.610579] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.626277] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.638796] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ 19.191684] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 20.224161] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.491474] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.512810] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.933704] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.999257] EDAC sbridge: Ver: 1.1.2 [ 26.074363] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 26.932051] NFS: Registering the id_resolver key type [ 26.939186] Key type id_resolver registered [ 26.943397] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg148-server login: [ 77.744755] hrtimer: interrupt took 5573959 ns [ 82.570265] libcfs: loading out-of-tree module taints kernel. [ 82.724250] Key type ._llcrypt registered [ 82.725808] Key type .llcrypt registered [ 82.910740] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 108.604404] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 110.243593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 110.286437] alg: No test for adler32 (adler32-zlib) [ 111.706721] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 112.425754] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 114.199239] Key type lgssc registered [ 116.141267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 132.834595] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 172.625344] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 187.022204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 187.068947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 188.305952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 188.355937] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 188.482332] Lustre: lustre-MDT0000: new disk, initializing [ 188.574202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.601083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 193.218328] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.383744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 205.519975] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 205.559816] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 205.564359] Lustre: Skipped 1 previous similar message [ 205.703462] Lustre: lustre-MDT0001: new disk, initializing [ 205.791398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 205.819206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 205.839764] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 209.885242] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.729687] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 225.539866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.082662] Lustre: lustre-OST0000: new disk, initializing [ 226.092990] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 226.102770] Lustre: 8447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 226.227927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 226.677334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 226.711199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 226.989277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 235.217123] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 251.216683] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 251.401215] Lustre: lustre-OST0001: new disk, initializing [ 251.405456] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 251.410716] Lustre: 9519:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 251.494111] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 257.307712] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 260.661928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 260.676925] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 260.786238] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 268.136554] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 276.574269] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 282.057792] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing check_logdir /tmp/testlogs/ [ 286.931373] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing yml_node [ 291.090755] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 293.600832] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 296.236910] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 298.161619] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 15 08:39:26 EDT 2026 [ 313.057918] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 314.310292] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 332.779109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.798197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.802920] Lustre: Skipped 1 previous similar message [ 332.818920] Lustre: Skipped 3 previous similar messages [ 335.258905] Lustre: server umount lustre-MDT0000 complete [ 338.321726] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789476007 with bad export cookie 15018687424332282938 [ 338.327738] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.334726] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 343.008171] LustreError: 8457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.009394] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.017185] LustreError: 8457:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 343.030447] Lustre: Skipped 2 previous similar messages [ 343.033562] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 345.069677] Lustre: server umount lustre-MDT0001 complete [ 348.951189] Lustre: server umount lustre-OST0000 complete [ 353.048428] Lustre: server umount lustre-OST0001 complete [ 359.182453] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 367.428171] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 397.425169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 397.712908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 397.740681] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 397.817859] Lustre: lustre-MDT0000: new disk, initializing [ 397.885125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 397.897850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 401.420534] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 413.051756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.205724] Lustre: 15070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 413.268828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 413.273138] Lustre: Skipped 1 previous similar message [ 413.401570] Lustre: lustre-MDT0001: new disk, initializing [ 413.535220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 413.603043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 413.619594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 419.738553] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 429.558185] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 435.276523] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 442.486671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 442.761633] Lustre: lustre-OST0000: new disk, initializing [ 442.765392] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 442.772690] Lustre: 17040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 442.858022] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 444.157372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 444.172394] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 444.298697] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 448.936699] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 459.151233] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 464.341667] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 464.547254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 469.422878] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 469.642750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 475.113689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 475.121394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.122335] LustreError: Skipped 1 previous similar message [ 475.122991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 475.122996] Lustre: Skipped 1 previous similar message [ 475.164306] Lustre: Skipped 2 previous similar messages [ 478.112626] Lustre: server umount lustre-OST0000 complete [ 485.355889] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.357967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 485.375474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 485.375483] Lustre: Skipped 1 previous similar message [ 488.267027] Lustre: server umount lustre-MDT0000 complete [ 490.470869] LustreError: 15938:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.494909] LustreError: 15938:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 491.894519] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789476160 with bad export cookie 15018687424332284471 [ 491.909026] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 491.911865] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 492.168669] Lustre: server umount lustre-MDT0001 complete [ 497.491830] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 08:42:45 (1789476165) [ 519.606757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 520.144933] LustreError: 19914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.270127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 525.263512] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 525.281771] LustreError: 19915:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.404060] LustreError: 19914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.497532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 539.602433] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 547.786402] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 554.178579] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 556.974790] Lustre: Failing over lustre-MDT0000 [ 559.075642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 559.077830] LustreError: 19911:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 559.095717] Lustre: Skipped 2 previous similar messages [ 559.117800] LustreError: 19911:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 559.436747] Lustre: server umount lustre-MDT0000 complete [ 575.455381] Lustre: 3644:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789476228/real 1789476228] req@ffff8aa9035c5c00 x1876401356694272/t0(0) o400->MGC192.168.201.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789476244 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 575.497826] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 575.519929] LustreError: 19909:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 575.538732] LustreError: 19909:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 580.325043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.831642] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd06d18a3d7f3c7bd [ 585.845474] Lustre: MGC192.168.201.148@tcp: Connection restored to 0@lo (at 0@lo) [ 586.144132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 586.154025] Lustre: Skipped 1 previous similar message [ 590.991734] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 591.338626] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 591.345386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 591.360042] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 599.923294] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 603.242536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.244354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 603.257496] Lustre: Skipped 1 previous similar message [ 603.273531] Lustre: Skipped 2 previous similar messages [ 609.546340] Lustre: server umount lustre-MDT0000 complete [ 609.760737] LustreError: 19915:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 609.768101] LustreError: 19915:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 612.520777] LustreError: 19894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789476281 with bad export cookie 15018687424332285885 [ 612.532190] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 612.856519] Lustre: server umount lustre-MDT0001 complete [ 620.957045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 626.613215] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 636.117805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 642.806326] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 650.990164] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 656.486315] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 658.591633] Lustre: Failing over lustre-MDT0000 [ 658.958914] Lustre: server umount lustre-MDT0000 complete [ 661.473345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 661.484542] Lustre: Skipped 1 previous similar message [ 677.796698] Lustre: 3645:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789476330/real 1789476330] req@ffff8aa903695c00 x1876401356723840/t0(0) o400->MGC192.168.201.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789476346 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 677.826902] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 677.852386] LustreError: 23250:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 677.886884] LustreError: 23250:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 678.284101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 687.409950] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 691.430766] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 692.716111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 692.718122] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 692.732792] Lustre: Skipped 1 previous similar message [ 692.783528] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 700.713171] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 707.373558] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 707.379793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 707.382855] Lustre: Skipped 1 previous similar message [ 707.401111] Lustre: Skipped 3 previous similar messages [ 710.590214] Lustre: server umount lustre-MDT0000 complete [ 713.825654] LustreError: 23231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789476382 with bad export cookie 15018687424332286823 [ 713.829902] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 713.842850] LustreError: 23231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 714.218347] Lustre: server umount lustre-MDT0001 complete [ 722.854935] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 08:46:30 (1789476390) [ 739.806524] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 748.839616] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 782.196930] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 792.536727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 792.815966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 792.841910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 792.935505] Lustre: lustre-MDT0000: new disk, initializing [ 793.051863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 793.055778] Lustre: Skipped 3 previous similar messages [ 793.074508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 797.682064] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 809.278747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 809.418926] Lustre: 29624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 809.785741] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 814.505682] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 826.908659] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 833.393822] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 841.275103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 841.613540] Lustre: 31595:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 843.146958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 843.212695] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 843.337109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 843.378897] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 849.677618] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 859.275278] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 865.104371] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 889.316021] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 889.332328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 889.363616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 889.386028] Lustre: Skipped 1 previous similar message [ 902.111273] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 902.489123] Lustre: server umount lustre-OST0000 complete [ 913.748269] Lustre: server umount lustre-MDT0000 complete [ 914.912256] LustreError: 30510:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 914.943349] LustreError: 30510:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 918.335987] LustreError: 29618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789476587 with bad export cookie 15018687424332287229 [ 918.353542] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 918.364561] LustreError: 29618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 919.003417] Lustre: server umount lustre-MDT0001 complete [ 931.246624] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 934.851493] Key type lgssc unregistered [ 935.347611] LNet: 33575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.354961] LNetError: 33575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 935.368719] LNet: Removed LNI 192.168.201.148@tcp [ 936.617512] Key type .llcrypt unregistered [ 936.619697] Key type ._llcrypt unregistered [ 967.045946] Key type ._llcrypt registered [ 967.050135] Key type .llcrypt registered [ 967.248244] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 968.929735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 969.245011] alg: No test for adler32 (adler32-zlib) [ 970.562371] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 970.911045] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 972.615206] Key type lgssc registered [ 973.918821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 981.739090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 981.755401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 983.248230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 987.604000] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 993.764750] LustreError: 34984:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 993.783287] LustreError: 34984:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 995.548217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 995.823189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1000.564156] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1009.721123] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1017.398379] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1026.076692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1026.610120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1033.722789] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1041.033385] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1041.544632] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1054.006645] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 08:52:02 (1789476722) [ 1055.988894] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1057.804721] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1064.508029] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 08:52:12 (1789476732) [ 1074.150189] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 08:52:22 (1789476742) [ 1075.326635] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1084.637459] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1084.643750] Lustre: Skipped 2 previous similar messages [ 1085.975953] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 08:52:34 (1789476754) [ 1089.752919] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1089.758703] Lustre: Skipped 3 previous similar messages [ 1109.288643] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1109.293879] Lustre: Skipped 6 previous similar messages [ 1112.388434] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 08:53:00 (1789476780) [ 1115.586266] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1120.010952] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1132.373601] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 08:53:19 (1789476799) [ 1135.742457] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1135.748935] Lustre: Skipped 3 previous similar messages [ 1152.244885] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 08:53:38 (1789476818) [ 1169.377482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1169.385966] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1169.392124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1169.405200] Lustre: Skipped 1 previous similar message [ 1173.847425] Lustre: server umount lustre-OST0000 complete [ 1179.620471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.622596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1179.630239] Lustre: Skipped 1 previous similar message [ 1179.642125] Lustre: Skipped 2 previous similar messages [ 1183.977618] Lustre: server umount lustre-MDT0000 complete [ 1184.741250] LustreError: 34983:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1184.767307] LustreError: 34983:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1188.545572] LustreError: 34963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789476857 with bad export cookie 6550348513738782924 [ 1188.549778] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1188.562055] LustreError: 34963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1189.065901] Lustre: server umount lustre-MDT0001 complete [ 1204.693338] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 1207.994717] Key type lgssc unregistered [ 1208.280596] LNet: 42843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1208.288530] LNetError: 42843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1208.322835] LNet: Removed LNI 192.168.201.148@tcp [ 1209.232395] Key type .llcrypt unregistered [ 1209.235586] Key type ._llcrypt unregistered [ 1231.148715] Key type ._llcrypt registered [ 1231.161990] Key type .llcrypt registered [ 1231.266876] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 1250.218922] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 1252.048186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1252.131334] alg: No test for adler32 (adler32-zlib) [ 1253.470260] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 1253.946198] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 1255.663288] Key type lgssc registered [ 1257.380705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1295.545897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1295.580337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1296.863459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1296.891394] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1296.985754] Lustre: lustre-MDT0000: new disk, initializing [ 1297.088635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1297.109280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1301.817126] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1313.995725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1314.092197] Lustre: 45995:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1314.141385] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1314.144591] Lustre: Skipped 1 previous similar message [ 1314.211188] Lustre: lustre-MDT0001: new disk, initializing [ 1314.299187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1314.335496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1314.348853] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1319.287857] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1329.797420] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1336.167404] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1343.322624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1343.703600] Lustre: lustre-OST0000: new disk, initializing [ 1343.707976] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1343.715798] Lustre: 47966:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1343.860442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.304868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1350.333333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1350.516859] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1351.560709] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1362.607165] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1366.894250] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1367.162586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1371.777648] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1372.002146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1376.225426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1376.228865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1376.236023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1376.245629] Lustre: Skipped 1 previous similar message [ 1376.268672] Lustre: Skipped 1 previous similar message [ 1380.186879] Lustre: server umount lustre-OST0000 complete [ 1385.952818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1385.969690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1386.001662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1391.599104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1391.602233] Lustre: Skipped 2 previous similar messages [ 1396.706038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1399.264276] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1399.496022] Lustre: server umount lustre-MDT0000 complete [ 1403.654466] LustreError: 45985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789477072 with bad export cookie 3375224039180506250 [ 1403.662622] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1403.669642] LustreError: 45985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1404.255892] Lustre: server umount lustre-MDT0001 complete [ 1415.129995] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 08:58:02 (1789477082) [ 1422.418848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1422.887063] LustreError: 50427:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1422.908772] LustreError: 50427:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1422.955330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1427.077092] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1428.450105] LustreError: 50428:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1432.558367] LustreError: 50427:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1435.618226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1435.966674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1441.020363] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1448.254949] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1453.243491] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1459.648386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1460.016687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1465.847669] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1472.999488] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1477.439039] LustreError: 52953:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1478.536331] Lustre: Modifying parameter general.timeout=1 in log params [ 1479.550986] Lustre: Modifying parameter general.timeout=2 in log params [ 1482.219610] Lustre: Modifying parameter general.timeout=5 in log params [ 1482.223220] Lustre: Skipped 2 previous similar messages [ 1486.593064] Lustre: Modifying parameter general.timeout=10 in log params [ 1486.600401] Lustre: Skipped 4 previous similar messages [ 1494.958581] Lustre: Modifying parameter general.timeout=20 in log params [ 1494.961899] Lustre: Skipped 9 previous similar messages [ 1511.686798] Lustre: Modifying parameter general.timeout=41 in log params [ 1511.690427] Lustre: Skipped 20 previous similar messages [ 1544.469964] Lustre: Modifying parameter general.timeout=74 in log params [ 1544.473448] Lustre: Skipped 32 previous similar messages [ 1602.422727] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:01:10 (1789477270) [ 1605.085601] Lustre: 59901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1606.726760] Lustre: 59999:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1609.421978] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 in log params [ 1609.427611] Lustre: Skipped 60 previous similar messages [ 1610.299405] Lustre: 60210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1610.310296] Lustre: 60210:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1639.410762] Lustre: 61782:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1639.418639] Lustre: 61782:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1639.433785] LustreError: 61782:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1643.570319] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:01:51 (1789477311) [ 1658.338746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1658.350974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1658.363406] Lustre: Skipped 3 previous similar messages [ 1658.376363] Lustre: Skipped 3 previous similar messages [ 1663.791549] Lustre: server umount lustre-MDT0000 complete [ 1667.939439] LustreError: 50407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789477336 with bad export cookie 3375224039180507132 [ 1667.946332] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1668.381652] Lustre: server umount lustre-MDT0001 complete [ 1683.144804] Lustre: server umount lustre-OST0000 complete [ 1712.806748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1713.076681] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1713.292527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1717.968559] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1727.019621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1727.151844] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1727.215411] Lustre: 64315:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1732.715550] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1741.784719] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1748.595187] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1757.334718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1757.553722] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1757.930772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1757.950579] Lustre: Skipped 1 previous similar message [ 1764.544784] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1771.853826] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1773.415550] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1773.422782] Lustre: Skipped 18 previous similar messages [ 1773.510580] Lustre: 66878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1783.780046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1783.795891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1783.816302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1783.833857] Lustre: Skipped 3 previous similar messages [ 1785.843239] Lustre: server umount lustre-OST0000 complete [ 1789.409321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1789.432457] Lustre: Skipped 1 previous similar message [ 1795.766806] Lustre: server umount lustre-MDT0000 complete [ 1798.815239] LustreError: 64305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789477467 with bad export cookie 3375224039180510093 [ 1798.826251] LustreError: 64305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1798.828771] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1799.176560] Lustre: server umount lustre-MDT0001 complete [ 1810.903477] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 1814.501630] Key type lgssc unregistered [ 1814.803644] LNet: 68087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.818098] LNetError: 68087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1814.846671] LNet: Removed LNI 192.168.201.148@tcp [ 1815.843753] Key type .llcrypt unregistered [ 1815.845992] Key type ._llcrypt unregistered [ 1840.795895] Key type ._llcrypt registered [ 1840.798937] Key type .llcrypt registered [ 1840.926870] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 1842.040372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1842.183276] alg: No test for adler32 (adler32-zlib) [ 1843.325921] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 1843.602703] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 1845.322306] Key type lgssc registered [ 1846.510483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1854.102558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1854.124986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1855.676661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1860.822310] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1865.191552] LustreError: 69498:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1865.209900] LustreError: 69498:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1869.293297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.126727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1874.853427] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1881.697937] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1887.495317] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1894.529599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1895.050371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1901.757394] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1908.973392] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1916.127169] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1926.114077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1926.122666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1926.137786] Lustre: Skipped 1 previous similar message [ 1926.144516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1926.158749] Lustre: Skipped 2 previous similar messages [ 1931.235632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1931.240745] Lustre: Skipped 1 previous similar message [ 1932.537047] Lustre: server umount lustre-MDT0000 complete [ 1935.917697] LustreError: 69477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789477604 with bad export cookie 7364322344786327296 [ 1935.920937] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1935.936899] LustreError: 69477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1936.432334] Lustre: server umount lustre-MDT0001 complete [ 1951.313325] Lustre: server umount lustre-OST0000 complete [ 1977.874751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1978.134941] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1978.186290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1978.409292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1983.387603] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1991.918965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1992.007774] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1992.041551] Lustre: 74372:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1992.116471] LustreError: 74384:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1992.483922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1997.461683] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2005.141824] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2010.902932] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2017.696000] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2017.816233] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2018.222626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2024.641306] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2032.989928] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2034.598545] Lustre: 76948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2036.669535] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2041.315922] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2041.319313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2041.323376] Lustre: Skipped 1 previous similar message [ 2041.327345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2046.952990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2046.961167] Lustre: Skipped 2 previous similar messages [ 2047.507847] Lustre: server umount lustre-OST0000 complete [ 2052.064944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2052.082071] Lustre: Skipped 1 previous similar message [ 2052.084688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2052.088373] Lustre: Skipped 1 previous similar message [ 2057.777218] Lustre: server umount lustre-MDT0000 complete [ 2061.517371] LustreError: 74366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789477730 with bad export cookie 7364322344786328143 [ 2061.522507] LustreError: 74366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2061.523587] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2061.940835] Lustre: server umount lustre-MDT0001 complete [ 2072.242239] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 2074.417472] Key type lgssc unregistered [ 2074.780808] LNet: 78160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2074.791594] LNetError: 78160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2074.817805] LNet: Removed LNI 192.168.201.148@tcp [ 2075.561811] Key type .llcrypt unregistered [ 2075.564386] Key type ._llcrypt unregistered [ 2099.298634] Key type ._llcrypt registered [ 2099.300777] Key type .llcrypt registered [ 2099.491748] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2100.609207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2100.896299] alg: No test for adler32 (adler32-zlib) [ 2101.864497] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 2102.052536] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 2103.687279] Key type lgssc registered [ 2104.954812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2111.647274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2111.661698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2113.086683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2116.944481] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2123.241445] LustreError: 79567:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2123.260121] LustreError: 79567:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2124.834665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2125.403383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2129.777451] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2137.583909] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2142.837746] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2148.855491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2149.208305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2154.937741] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2162.608566] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2169.711242] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2170.759468] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2172.698207] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2172.704361] Lustre: Skipped 1 previous similar message [ 2175.159614] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2175.163071] Lustre: Skipped 2 previous similar messages [ 2179.548740] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2179.551606] Lustre: Skipped 4 previous similar messages [ 2187.551920] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2187.556457] Lustre: Skipped 9 previous similar messages [ 2203.685196] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2203.691308] Lustre: Skipped 18 previous similar messages [ 2235.773595] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2235.779765] Lustre: Skipped 34 previous similar messages [ 2266.565668] Lustre: 87668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2269.283966] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 09:12:17 (1789477937) [ 2270.756168] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2272.514822] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 09:12:20 (1789477940) [ 2278.368647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2278.384449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2278.393033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2281.441512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2281.443839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2281.461417] Lustre: Skipped 1 previous similar message [ 2282.787493] Lustre: server umount lustre-OST0000 complete [ 2286.561070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2286.562192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2286.580308] Lustre: Skipped 1 previous similar message [ 2291.680404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2291.686605] Lustre: Skipped 2 previous similar messages [ 2296.806087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2300.383112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2300.590215] Lustre: server umount lustre-MDT0000 complete [ 2301.924356] LustreError: 79561:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2301.947395] LustreError: 79561:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2304.471808] LustreError: 79548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789477973 with bad export cookie 12738975374984005601 [ 2304.480223] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2304.505856] LustreError: 79548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2305.215282] Lustre: server umount lustre-MDT0001 complete [ 2315.520545] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 2317.973491] Key type lgssc unregistered [ 2318.276058] LNet: 88918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2318.281663] LNetError: 88918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2318.292215] LNet: Removed LNI 192.168.201.148@tcp [ 2319.081056] Key type .llcrypt unregistered [ 2319.084732] Key type ._llcrypt unregistered [ 2323.654576] Key type ._llcrypt registered [ 2323.661430] Key type .llcrypt registered [ 2323.794906] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2328.454466] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules [ 2329.668310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2329.721921] alg: No test for adler32 (adler32-zlib) [ 2330.756739] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 2330.988579] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 2332.671261] Key type lgssc registered [ 2334.065989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2335.067205] LustreError: 90119:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2353.295450] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2356.681019] LustreError: 90119:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2362.570291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2362.593959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2364.072097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2368.419888] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2373.605132] LustreError: 91111:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2373.633514] LustreError: 91111:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2376.560186] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:14:04 (1789478044) [ 2378.735165] LustreError: 91112:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.193755] LustreError: 91143:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2381.202436] LustreError: 91143:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2381.206969] LustreError: 91143:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2381.477539] Lustre: server umount lustre-MDT0000 complete [ 2395.503401] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 2398.088227] Key type lgssc unregistered [ 2398.413844] LNet: 92636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2398.422850] LNetError: 92636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2398.453783] LNet: Removed LNI 192.168.201.148@tcp [ 2399.390722] Key type .llcrypt unregistered [ 2399.397140] Key type ._llcrypt unregistered [ 2416.604391] Key type ._llcrypt registered [ 2416.606388] Key type .llcrypt registered [ 2416.717351] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2417.747469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2417.933180] alg: No test for adler32 (adler32-zlib) [ 2418.978522] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 2419.171588] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 2420.815221] Key type lgssc registered [ 2421.842615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2428.848886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2428.905884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2430.679176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2435.200786] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2440.178977] LustreError: 93850:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.206850] LustreError: 93850:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2443.853557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2444.498515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2448.722696] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2456.189768] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2462.699314] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2468.792463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2469.498657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2475.513975] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2484.158506] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2526.251468] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:16:34 (1789478194) [ 2527.928482] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2529.785534] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:16:37 (1789478197) [ 2536.420100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2536.426367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2536.443683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2537.442786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2537.446549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2537.456896] Lustre: Skipped 1 previous similar message [ 2537.466305] Lustre: Skipped 1 previous similar message [ 2539.625918] Lustre: server umount lustre-MDT0000 complete [ 2542.563632] LustreError: 93849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2542.579090] LustreError: 93849:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2542.808100] LustreError: 93829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789478211 with bad export cookie 11838461224014463808 [ 2542.810944] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2542.816768] LustreError: 93829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2543.076386] Lustre: server umount lustre-MDT0001 complete [ 2557.677139] Lustre: server umount lustre-OST0000 complete [ 2569.047603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2569.361867] LustreError: 98124:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2569.518229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2574.102029] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2574.818671] LustreError: 98125:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.940675] LustreError: 98124:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2582.970233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.901187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2588.737927] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2597.819849] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2605.920655] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2621.527815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2621.802373] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2621.812712] LustreError: 98113:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2621.851586] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2621.861619] LustreError: 100208:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2621.870796] LustreError: 100208:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2621.877824] LustreError: 100208:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2621.955753] Lustre: server umount lustre-OST0000 complete [ 2621.960634] LustreError: 100208:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2631.791176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2632.004288] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2632.017203] LustreError: 98114:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2632.077075] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2632.087492] LustreError: 100676:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2632.102576] LustreError: 100676:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2632.109931] LustreError: 100676:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2632.208974] Lustre: server umount lustre-OST0000 complete [ 2632.212637] LustreError: 100676:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2642.369696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2642.504111] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2642.508574] Lustre: Found index 0 for lustre-OST0000, updating log [ 2642.511366] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2642.569911] Lustre: lustre-OST0000: new disk, initializing [ 2642.573666] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2642.577302] Lustre: 101211:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2642.742626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2649.658208] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2653.855846] Lustre: Failing over lustre-OST0000 [ 2653.951705] Lustre: server umount lustre-OST0000 complete [ 2660.321539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2660.325847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2660.382184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2660.396354] Lustre: Skipped 1 previous similar message [ 2663.494963] Lustre: server umount lustre-MDT0000 complete [ 2665.441546] LustreError: 98124:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2665.451919] LustreError: 98124:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2666.979418] LustreError: 98106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789478335 with bad export cookie 11838461224014466552 [ 2666.988349] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2666.994150] LustreError: 98106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2667.036763] LustreError: 93161:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting [ 2667.516733] Lustre: server umount lustre-MDT0001 complete [ 2677.042651] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:19:04 (1789478344) [ 2697.279132] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2708.246880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2708.616133] LustreError: 103554:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2708.733495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2713.215973] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2721.988997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2726.970935] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2729.540795] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2736.508523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2744.206983] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2755.855579] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2756.028877] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2756.187281] Lustre: lustre-OST0001: new disk, initializing [ 2756.191645] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2756.203308] Lustre: 105877:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2756.591992] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2756.601208] Lustre: Skipped 2 previous similar messages [ 2763.669652] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2766.525348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2766.537162] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2766.754118] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2773.288246] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2782.294405] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2787.198079] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.48@tcp (stopping) [ 2787.297738] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2787.317076] Lustre: Skipped 3 previous similar messages [ 2792.296389] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.48@tcp (stopping) [ 2792.299155] Lustre: Skipped 3 previous similar messages [ 2792.799382] Lustre: server umount lustre-MDT0001 complete [ 2797.420123] LustreError: 103551:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2797.461931] LustreError: 103551:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2802.517424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2802.693410] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2802.698207] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2802.746921] Lustre: 103544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2804.472831] LustreError: 107448:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2804.485622] LustreError: 107448:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2804.502851] LustreError: 103546:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2804.669687] LustreError: 107459:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.148@tcp: cfg command failed: rc = -17 [ 2804.683500] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2804.683500] [ 2808.551629] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2808.807245] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2820.933483] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 09:21:28 (1789478488) [ 2841.952247] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2860.743086] Lustre: 109490:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2886.199757] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2890.339928] Lustre: 110626:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2970.593510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2970.595484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2970.603823] Lustre: Skipped 2 previous similar messages [ 2970.620920] Lustre: Skipped 6 previous similar messages [ 2977.013271] Lustre: server umount lustre-MDT0000 complete [ 2980.837073] LustreError: 103550:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2980.853340] LustreError: 103550:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 2980.931685] LustreError: 103534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789478649 with bad export cookie 11838461224014467546 [ 2980.934540] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2980.942969] LustreError: 103534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2981.197902] Lustre: server umount lustre-MDT0001 complete [ 2995.376956] Lustre: server umount lustre-OST0000 complete [ 3009.434489] Lustre: server umount lustre-OST0001 complete [ 3016.155487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3026.468350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3035.160497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3044.052115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3062.581323] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 3073.160967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3073.185507] Lustre: lustre-MDT0000: reset Object Index mappings [ 3073.856954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3073.864226] Lustre: Skipped 1 previous similar message [ 3078.689583] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3086.780235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3086.798573] Lustre: lustre-MDT0001: reset Object Index mappings [ 3092.177517] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3095.302440] Lustre: 115744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3101.961136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3109.483655] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3112.368326] LustreError: 116099:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3112.400219] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3112.402319] LustreError: 116099:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3118.613902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3119.596709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3125.802043] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3125.818171] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3129.304699] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3138.706035] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3143.125054] Lustre: 117621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3190.099428] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:27:37 (1789478857) [ 3217.893039] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3217.897345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3217.918019] Lustre: Skipped 3 previous similar messages [ 3217.941348] Lustre: Skipped 7 previous similar messages [ 3222.055360] Lustre: server umount lustre-MDT0000 complete [ 3226.465061] LustreError: 114577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789478895 with bad export cookie 11838461224014514411 [ 3226.466840] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3226.473363] LustreError: 114577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3233.520563] Lustre: server umount lustre-MDT0001 complete [ 3238.752184] LustreError: 118378:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000400:0x2d:0x0].0x0 (ffff8aa903732b00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3245.307837] Lustre: server umount lustre-OST0000 complete [ 3249.642774] LustreError: 118581:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000400:0x76:0x0].0x0 (ffff8aaa3b40d100) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3256.039448] Lustre: server umount lustre-OST0001 complete [ 3262.257630] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 3273.876989] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 3321.469965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3321.912285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3321.972065] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3322.148197] Lustre: lustre-MDT0000: new disk, initializing [ 3322.318537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3322.321469] Lustre: Skipped 3 previous similar messages [ 3322.377527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3328.333661] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3335.395824] Lustre: server umount lustre-MDT0000 complete [ 3346.048469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3346.530792] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3346.551277] Lustre: 121892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3346.561757] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3352.322203] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3359.538486] Lustre: server umount lustre-MDT0000 complete [ 3370.258519] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:30:37 (1789479037) [ 3371.891230] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3373.871985] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 09:30:41 (1789479041) [ 3393.457898] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 3404.041228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3408.224610] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3417.514588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3417.693758] Lustre: 123941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3417.800277] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3417.807713] Lustre: Skipped 1 previous similar message [ 3417.999230] Lustre: lustre-MDT0001: new disk, initializing [ 3418.259842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3418.283318] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3425.588770] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3431.629538] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3439.225449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3439.654883] Lustre: lustre-OST0000: new disk, initializing [ 3439.660777] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3439.670597] Lustre: 125522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3441.128120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3441.139755] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3441.308361] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3447.275939] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3457.957875] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3458.137099] Lustre: lustre-OST0001: new disk, initializing [ 3458.140137] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3458.145394] Lustre: 126394:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3459.721316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3459.732592] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3459.826333] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3465.630171] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3475.476659] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3479.731368] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3845.969113] Lustre: 132803:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3845.985188] Lustre: 132803:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3845.998949] Lustre: 132803:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3846.008602] Lustre: 132803:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3846.014698] Lustre: 132803:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3846.020406] Lustre: 132803:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4110.969039] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:42:57 (1789479777) [ 4129.761743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4129.778650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4129.784788] Lustre: Skipped 3 previous similar messages [ 4129.787909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4129.790379] Lustre: Skipped 4 previous similar messages [ 4132.637144] Lustre: server umount lustre-MDT0000 complete [ 4135.911497] LustreError: 123949:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4135.933194] LustreError: 123949:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [ 4136.903030] LustreError: 132828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789479805 with bad export cookie 11838461224014581065 [ 4136.916839] LustreError: 132828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4136.917210] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4141.026472] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4141.036497] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4141.041096] Lustre: Skipped 4 previous similar messages [ 4141.059944] Lustre: Skipped 5 previous similar messages [ 4151.263213] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4151.746335] Lustre: server umount lustre-MDT0001 complete [ 4162.653949] Lustre: server umount lustre-OST0000 complete [ 4172.179645] Lustre: server umount lustre-OST0001 complete [ 4177.611069] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 4187.884205] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 4232.683607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4233.224459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4233.296180] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4233.454518] Lustre: lustre-MDT0000: new disk, initializing [ 4233.557136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4233.573913] Lustre: Skipped 5 previous similar messages [ 4233.616966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4239.044984] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4250.445445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4250.623511] Lustre: 135857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4250.710752] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4250.718307] Lustre: Skipped 1 previous similar message [ 4250.840632] Lustre: lustre-MDT0001: new disk, initializing [ 4250.975308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4250.991901] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4257.934915] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4273.369183] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4281.103426] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4288.592561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4288.910379] Lustre: lustre-OST0000: new disk, initializing [ 4288.917862] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4288.923822] Lustre: 137826:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4289.006608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4289.016831] Lustre: Skipped 1 previous similar message [ 4290.122040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4290.128241] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4290.241492] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4295.353051] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4304.274294] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4314.813535] Lustre: lustre-MDD0000: changelog on [ 4318.011487] Lustre: lustre-MDD0001: changelog on [ 5879.707432] Lustre: lustre-MDD0001: changelog off [ 5881.435256] Lustre: 137419:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5882.173150] Lustre: lustre-MDD0000: changelog off [ 5887.974649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5887.979227] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5887.979826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5887.979831] Lustre: Skipped 4 previous similar messages [ 5887.986491] Lustre: Skipped 1 previous similar message [ 5888.012190] LustreError: Skipped 1 previous similar message [ 5892.501774] Lustre: server umount lustre-OST0000 complete [ 5898.210151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5898.216289] Lustre: Skipped 1 previous similar message [ 5898.218586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5898.221658] Lustre: Skipped 2 previous similar messages [ 5902.074315] Lustre: server umount lustre-MDT0000 complete [ 5903.334058] LustreError: 135868:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5903.346375] LustreError: 135868:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 5904.902280] LustreError: 135847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789481573 with bad export cookie 11838461224015394486 [ 5904.915456] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5905.127501] Lustre: server umount lustre-MDT0001 complete [ 5913.291364] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 5915.446930] Key type lgssc unregistered [ 5915.632503] LNet: 146916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5915.640544] LNetError: 146916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5915.653924] LNet: Removed LNI 192.168.201.148@tcp [ 5916.265119] Key type .llcrypt unregistered [ 5916.266212] Key type ._llcrypt unregistered [ 5931.725893] Key type ._llcrypt registered [ 5931.727101] Key type .llcrypt registered [ 5931.824899] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 5942.888261] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 5943.637990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5943.673350] alg: No test for adler32 (adler32-zlib) [ 5944.645523] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 5944.859441] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 5946.519208] Key type lgssc registered [ 5947.268700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5977.794249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5977.826568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5979.018103] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5979.043544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5979.123263] Lustre: lustre-MDT0000: new disk, initializing [ 5979.210666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5979.224885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5982.837722] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5990.837836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5990.936328] Lustre: 150093:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5990.974740] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5990.978612] Lustre: Skipped 1 previous similar message [ 5991.042813] Lustre: lustre-MDT0001: new disk, initializing [ 5991.107509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5991.145475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5991.158722] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5994.711379] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6002.234792] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6007.737598] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6014.313000] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6014.580057] Lustre: lustre-OST0000: new disk, initializing [ 6014.585408] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6014.594808] Lustre: 152058:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6014.685277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6019.656891] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6021.682448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6021.689441] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6021.765494] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6026.958495] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6031.391712] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_module obdecho/obdecho [ 6033.091457] LustreError: 153060:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6033.107379] LustreError: 153060:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6034.888802] LustreError: 153156:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6047.713065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6047.715904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6047.725040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6047.738297] Lustre: Skipped 1 previous similar message [ 6051.023916] Lustre: server umount lustre-MDT0000 complete [ 6054.335683] LustreError: 150084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789481723 with bad export cookie 9539268345992954364 [ 6054.349093] LustreError: 150084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6054.356843] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6054.784471] Lustre: server umount lustre-MDT0001 complete [ 6068.954809] Lustre: server umount lustre-OST0000 complete [ 6087.435530] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 6099.188260] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 6150.387105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6150.827925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6150.874320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6151.016194] Lustre: lustre-MDT0000: new disk, initializing [ 6151.162649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6151.182971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6157.405399] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6169.632494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6169.819398] Lustre: 156876:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6169.932698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6169.941702] Lustre: Skipped 1 previous similar message [ 6170.066217] Lustre: lustre-MDT0001: new disk, initializing [ 6170.191971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6170.267794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6170.305085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6175.822235] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6187.022718] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6194.817963] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6204.359869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6204.904226] Lustre: lustre-OST0000: new disk, initializing [ 6204.907853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6204.923561] Lustre: 158847:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6205.083389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6210.833386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6210.864311] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6211.099229] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6213.800733] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6226.206722] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6232.060438] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6232.393857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6237.291887] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6237.624712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6241.760177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6241.763124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6241.771332] LustreError: Skipped 1 previous similar message [ 6241.781312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6241.797820] Lustre: Skipped 2 previous similar messages [ 6241.825255] Lustre: Skipped 3 previous similar messages [ 6245.926583] Lustre: server umount lustre-OST0000 complete [ 6252.001936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6252.003178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6252.019184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6252.036767] Lustre: Skipped 1 previous similar message [ 6256.089248] Lustre: server umount lustre-MDT0000 complete [ 6257.120285] LustreError: 156884:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6257.138375] LustreError: 156884:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6259.789068] LustreError: 156868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789481928 with bad export cookie 9539268345992955323 [ 6259.793076] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6259.804021] LustreError: 156868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6260.319453] Lustre: server umount lustre-MDT0001 complete [ 6264.462586] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:18:52 (1789481932) [ 6265.988787] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6267.983798] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:18:55 (1789481935) [ 6276.224356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6276.642730] LustreError: 161261:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6276.730261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6281.809719] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6282.223288] LustreError: 161262:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6286.316239] LustreError: 161261:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6290.487122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6295.301779] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6303.357897] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6310.894640] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6319.612830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6319.946786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6319.952667] Lustre: Skipped 1 previous similar message [ 6327.259821] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6335.303534] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6345.698917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6345.702162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6345.708818] Lustre: Skipped 3 previous similar messages [ 6345.728275] Lustre: Skipped 2 previous similar messages [ 6348.231756] Lustre: server umount lustre-MDT0000 complete [ 6350.818485] LustreError: 161262:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6350.838392] LustreError: 161262:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6350.998117] LustreError: 161242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482020 with bad export cookie 9539268345992956205 [ 6351.018225] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6351.018463] LustreError: 161242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6351.750341] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.48@tcp (stopping) [ 6351.759220] Lustre: Skipped 1 previous similar message [ 6355.940349] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6357.525751] Lustre: server umount lustre-MDT0001 complete [ 6364.945056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6378.354290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6378.684472] LustreError: 165106:0:(llog.c:1655:llog_backup()) MGC192.168.201.148@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6378.693996] Lustre: 165106:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.148@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6381.485726] LustreError: 165111:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6381.507328] LustreError: 165111:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6397.797841] LustreError: 165111:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6397.824537] LustreError: 165111:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 6431.715175] LustreError: 165113:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6431.735753] LustreError: 165113:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 25 previous similar messages [ 6439.391443] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6439.395626] Lustre: 165106:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.148@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6480.386417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6485.483087] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6485.770467] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6495.724995] LustreError: 165113:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6495.744930] LustreError: 165113:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 42 previous similar messages [ 6495.936958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6502.381742] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6502.391077] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6511.404782] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6512.897583] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6519.697554] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6521.531511] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6543.329616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6543.330757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6543.341477] Lustre: Skipped 2 previous similar messages [ 6543.353333] Lustre: Skipped 5 previous similar messages [ 6546.289669] Lustre: server umount lustre-MDT0000 complete [ 6550.395564] LustreError: 161242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482219 with bad export cookie 9539268345992962659 [ 6550.407321] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6550.422440] LustreError: 161242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6550.935338] Lustre: server umount lustre-MDT0001 complete [ 6565.954080] Lustre: server umount lustre-OST0000 complete [ 6590.305283] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 6602.336749] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 6640.551908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6640.838195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6640.857891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6640.950512] Lustre: lustre-MDT0000: new disk, initializing [ 6641.033558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6641.039390] Lustre: Skipped 1 previous similar message [ 6641.053580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6645.892324] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6657.695396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6657.884803] Lustre: 170212:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6657.969217] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6658.220716] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6662.915668] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6673.315862] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6680.525892] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6688.273221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6688.668934] Lustre: lustre-OST0000: new disk, initializing [ 6688.674851] Lustre: Skipped 1 previous similar message [ 6688.686200] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6688.696725] Lustre: Skipped 2 previous similar messages [ 6688.706653] Lustre: 172181:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6695.233322] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6699.091032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6699.102255] Lustre: Skipped 1 previous similar message [ 6699.112090] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6699.240851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6706.198093] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6714.279819] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6714.495399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6719.393951] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6719.741720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6725.094736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6725.097089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6725.100238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6725.107802] LustreError: Skipped 1 previous similar message [ 6725.137017] Lustre: Skipped 1 previous similar message [ 6735.839145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6736.077576] Lustre: server umount lustre-OST0000 complete [ 6746.514369] Lustre: server umount lustre-MDT0000 complete [ 6749.785494] LustreError: 172180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482418 with bad export cookie 9539268345992979494 [ 6749.789726] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6749.805408] LustreError: 172180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6750.173640] Lustre: server umount lustre-MDT0001 complete [ 6758.549448] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:27:06 (1789482426) [ 6765.089773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6765.531382] LustreError: 174597:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6765.554242] LustreError: 174597:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 6770.283743] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6779.672331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6785.739020] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6794.063169] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6800.297097] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6808.958023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6818.148982] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6826.308905] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6838.842283] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:28:26 (1789482506) [ 6846.438921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6846.447507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6846.450488] LustreError: Skipped 1 previous similar message [ 6846.454265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6846.454274] Lustre: Skipped 9 previous similar messages [ 6846.494821] Lustre: Skipped 3 previous similar messages [ 6849.633949] Lustre: server umount lustre-OST0000 complete [ 6861.192458] Lustre: server umount lustre-MDT0000 complete [ 6866.776534] LustreError: 175332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482535 with bad export cookie 9539268345992980376 [ 6866.819900] LustreError: 175332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6866.820381] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6867.610634] Lustre: server umount lustre-MDT0001 complete [ 6882.189662] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 6885.767893] Key type lgssc unregistered [ 6886.099513] LNet: 178326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6886.105899] LNetError: 178326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6887.149394] LNet: Removed LNI 192.168.201.148@tcp [ 6888.151846] Key type .llcrypt unregistered [ 6888.155325] Key type ._llcrypt unregistered [ 6907.430957] Key type ._llcrypt registered [ 6907.435543] Key type .llcrypt registered [ 6907.517253] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 6908.302866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6908.430507] alg: No test for adler32 (adler32-zlib) [ 6909.482566] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 6909.689337] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 6911.287300] Key type lgssc registered [ 6912.348948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6919.825277] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6919.847726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6935.455382] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6940.575779] LustreError: 179567:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.148@tcp: failed processing log, type 4: rc = -110 [ 6945.695428] LustreError: 179567:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6945.707901] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6945.713751] LustreError: MGC192.168.201.148@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6945.724970] LustreError: 179567:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6945.741667] LustreError: 179567:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 6945.749466] LustreError: 179567:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 6945.829860] Lustre: server umount lustre-OST0000 complete [ 6945.842361] LustreError: 179567:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6955.596631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6957.047199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6961.629168] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6967.276910] LustreError: 180066:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6967.315189] LustreError: 180066:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6970.358791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6970.842449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6975.066800] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6982.253480] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6986.943076] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6994.110334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6994.403133] LustreError: 181882:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6994.420138] Lustre: 181882:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.148@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6994.672825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7001.015941] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7007.735769] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7011.808991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7011.822375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7011.832358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7013.345564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7013.350344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7013.363208] Lustre: Skipped 1 previous similar message [ 7013.387141] Lustre: Skipped 2 previous similar messages [ 7016.732382] Lustre: server umount lustre-MDT0000 complete [ 7018.468339] LustreError: 180063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7018.484627] LustreError: 180063:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7020.384854] LustreError: 180048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482689 with bad export cookie 17534557569003407046 [ 7020.389070] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7020.401327] LustreError: 180048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7020.943701] Lustre: server umount lustre-MDT0001 complete [ 7034.943949] Lustre: server umount lustre-OST0000 complete [ 7043.789742] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 7054.673829] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 7094.734960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7095.085832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7095.136819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7095.276590] Lustre: lustre-MDT0000: new disk, initializing [ 7095.378144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7095.396579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7100.041361] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7111.569279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7111.775245] Lustre: 185407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7111.900383] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7111.915468] Lustre: Skipped 1 previous similar message [ 7112.076750] Lustre: lustre-MDT0001: new disk, initializing [ 7112.274936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7112.321247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7112.331349] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7117.716096] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7127.532798] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7133.807465] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7140.642593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7141.032392] Lustre: lustre-OST0000: new disk, initializing [ 7141.040546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7141.047650] Lustre: 187378:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7141.200069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7148.651567] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7149.645688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7149.673086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7149.947557] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7159.202656] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7164.633748] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7164.855223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7170.425473] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7170.727617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7175.662658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7175.700692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7175.705318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7178.776524] Lustre: server umount lustre-OST0000 complete [ 7183.840819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7183.852896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7183.865800] Lustre: Skipped 1 previous similar message [ 7183.878316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7183.901802] Lustre: Skipped 1 previous similar message [ 7190.026164] Lustre: server umount lustre-MDT0000 complete [ 7191.033955] LustreError: 186215:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7191.044083] LustreError: 186215:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7194.335080] LustreError: 185401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482863 with bad export cookie 17534557569003407767 [ 7194.340153] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7194.343944] LustreError: 185401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7194.673561] Lustre: server umount lustre-MDT0001 complete [ 7204.625964] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:34:32 (1789482872) [ 7220.875519] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 7223.449260] Key type lgssc unregistered [ 7223.726603] LNet: 190323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7223.734614] LNetError: 190323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7223.752571] LNet: Removed LNI 192.168.201.148@tcp [ 7224.631137] Key type .llcrypt unregistered [ 7224.638202] Key type ._llcrypt unregistered [ 7241.212437] Key type ._llcrypt registered [ 7241.220691] Key type .llcrypt registered [ 7241.297697] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 7242.406863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7242.468508] alg: No test for adler32 (adler32-zlib) [ 7243.480964] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 7243.684916] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 7245.341763] Key type lgssc registered [ 7246.473776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7253.758986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7253.782133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7255.122934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7259.547671] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7265.257414] LustreError: 191536:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7265.284775] LustreError: 191536:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7268.079117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7268.443023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7273.074144] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7279.863928] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7284.778663] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7291.660409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7291.793834] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7291.795854] LustreError: 193398:0:(llog.c:1696:llog_backup()) MGC192.168.201.148@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7291.802568] Lustre: 193398:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.148@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7291.936977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7296.993889] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7303.532532] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7309.287133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7309.295879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7309.325125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7310.305125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7310.308749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7310.329103] Lustre: Skipped 1 previous similar message [ 7310.342385] Lustre: Skipped 2 previous similar messages [ 7311.926731] Lustre: server umount lustre-MDT0000 complete [ 7315.426178] LustreError: 191533:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7315.460027] LustreError: 191533:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7315.624685] LustreError: 191516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482984 with bad export cookie 1389986970491371870 [ 7315.651702] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7315.869020] Lustre: server umount lustre-MDT0001 complete [ 7329.876559] Lustre: server umount lustre-OST0000 complete [ 7339.955824] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 7351.964902] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 7386.144579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7386.418857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7386.448055] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7386.507516] Lustre: lustre-MDT0000: new disk, initializing [ 7386.570988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7386.591380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7391.099399] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7400.977870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7401.090889] Lustre: 196927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7401.122537] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7401.125800] Lustre: Skipped 1 previous similar message [ 7401.129515] LustreError: 197708:0:(llog.c:1696:llog_backup()) MGC192.168.201.148@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7401.136874] LustreError: 197708:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7401.142747] Lustre: 197708:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.148@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7401.169782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7401.174122] Lustre: Skipped 1 previous similar message [ 7401.228678] Lustre: lustre-MDT0001: new disk, initializing [ 7401.344610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7401.369278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7401.386609] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7406.334236] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7417.146356] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7423.883158] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7432.063767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7432.362864] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7432.364965] Lustre: Skipped 1 previous similar message [ 7432.372535] LustreError: 198871:0:(llog.c:1696:llog_backup()) MGC192.168.201.148@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7432.386213] LustreError: 198871:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7432.395293] Lustre: 198871:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.148@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7432.417211] Lustre: 198871:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7432.473359] Lustre: lustre-OST0000: new disk, initializing [ 7432.477591] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7432.483940] Lustre: 198894:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7432.571765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7438.397392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7438.415497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7438.599647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7439.550493] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7448.314801] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7452.213752] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7452.386552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7456.561891] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7456.846931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7459.299826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7459.300887] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7459.319534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7459.331623] Lustre: Skipped 1 previous similar message [ 7464.416800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7464.426480] Lustre: Skipped 1 previous similar message [ 7469.545122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7469.550235] Lustre: Skipped 1 previous similar message [ 7473.119938] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7473.395388] Lustre: server umount lustre-OST0000 complete [ 7478.243821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7478.249733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7478.271614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7478.278325] Lustre: Skipped 1 previous similar message [ 7483.851660] Lustre: server umount lustre-MDT0000 complete [ 7484.896571] LustreError: 196937:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7487.557827] LustreError: 196917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483156 with bad export cookie 1389986970491372591 [ 7487.558737] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7487.572819] LustreError: 196917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7487.834236] Lustre: server umount lustre-MDT0001 complete [ 7495.995534] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:39:24 (1789483164) [ 7510.769856] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 7521.066518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7521.512035] LustreError: 202020:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7521.541745] LustreError: 202020:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7521.553084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7525.743710] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7526.882101] LustreError: 202021:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7533.619617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7539.139727] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7541.989581] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7548.097277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7555.060675] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7564.364069] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7564.535805] Lustre: lustre-OST0001: new disk, initializing [ 7564.541509] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7564.547821] Lustre: 204327:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7564.652425] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7564.654508] Lustre: Skipped 2 previous similar messages [ 7570.940946] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7573.620933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7573.644433] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7573.751683] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7582.126140] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7589.250836] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7591.155351] Lustre: DEBUG MARKER: ADD OST3 [ 7596.096585] loop: module loaded [ 7596.177220] loop0: detected capacity change from 0 to 204800000 [ 7596.376225] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7596.396461] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7596.420102] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7603.153769] loop0: detected capacity change from 0 to 204800000 [ 7603.419619] Lustre: DEBUG MARKER: STOP OST3 [ 7609.337726] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7609.595396] Lustre: lustre-OST0003: new disk, initializing [ 7609.599577] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7609.607054] Lustre: 206368:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7611.822368] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7612.023016] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7612.034094] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7612.087037] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7613.410298] Lustre: Failing over lustre-OST0003 [ 7613.645994] Lustre: server umount lustre-OST0003 complete [ 7613.857193] LustreError: 206526:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7615.904482] LustreError: 204794:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7615.938553] LustreError: 204794:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7617.020366] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7617.038786] Lustre: Skipped 2 previous similar messages [ 7617.053092] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7618.277875] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7629.691904] Lustre: DEBUG MARKER: START OST3 again [ 7632.368079] LustreError: 204794:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7632.411670] LustreError: 204794:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 7636.185775] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7636.367185] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7636.373761] Lustre: Skipped 1 previous similar message [ 7636.382310] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7637.652704] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7637.798220] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7637.798323] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7642.393593] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7662.499347] Lustre: Failing over lustre-OST0003 [ 7662.601547] Lustre: server umount lustre-OST0003 complete [ 7663.585256] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7663.591395] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7663.591783] LustreError: Skipped 1 previous similar message [ 7663.606191] Lustre: Skipped 1 previous similar message [ 7667.189732] LustreError: 208165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7667.202422] LustreError: 208165:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7682.528519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7682.553114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7682.554744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7682.559330] Lustre: Skipped 3 previous similar messages [ 7686.678676] Lustre: server umount lustre-MDT0000 complete [ 7690.392622] LustreError: 202002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483359 with bad export cookie 1389986970491373473 [ 7690.393419] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7690.402050] LustreError: 202002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7697.040317] Lustre: server umount lustre-MDT0001 complete [ 7701.313885] Lustre: server umount lustre-OST0000 complete [ 7705.368283] Lustre: server umount lustre-OST0001 complete [ 7712.280664] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 7721.295390] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 7758.514166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7758.718737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7758.742643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7758.806843] Lustre: lustre-MDT0000: new disk, initializing [ 7758.871988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7764.317923] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7776.226042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7776.418101] Lustre: 211943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7776.479906] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7776.486501] Lustre: Skipped 1 previous similar message [ 7776.538773] Lustre: lustre-MDT0001: new disk, initializing [ 7776.623709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7776.630852] Lustre: Skipped 1 previous similar message [ 7776.679570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7776.693945] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7781.873708] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7793.642744] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7799.212939] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7806.609454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7806.868807] Lustre: 213909:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7808.190316] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7808.329149] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7813.286750] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7822.321566] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7827.110733] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7827.321571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7832.455894] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7832.671774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7839.202180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7839.206578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7839.212785] Lustre: Skipped 5 previous similar messages [ 7839.220846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7839.220864] Lustre: Skipped 6 previous similar messages [ 7841.020795] Lustre: server umount lustre-OST0000 complete [ 7859.167424] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7859.426776] Lustre: server umount lustre-MDT0000 complete [ 7859.682509] LustreError: 211953:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7859.702995] LustreError: 211953:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 7863.057975] LustreError: 211936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483532 with bad export cookie 1389986970491375769 [ 7863.063048] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7863.068562] LustreError: 211936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7863.486172] Lustre: server umount lustre-MDT0001 complete [ 7867.068658] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 10:45:35 (1789483535) [ 7883.582585] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 7892.819200] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 7928.117696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7928.493735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7928.539230] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7928.544392] Lustre: Skipped 1 previous similar message [ 7928.684857] Lustre: lustre-MDT0000: new disk, initializing [ 7928.688930] Lustre: Skipped 1 previous similar message [ 7928.782405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7928.788219] Lustre: Skipped 1 previous similar message [ 7933.134950] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7944.533494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7944.675754] Lustre: 218521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7944.901771] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7949.565383] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7960.103250] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7966.793427] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7975.382276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7975.590482] Lustre: 220488:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7977.504151] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7977.635681] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7982.754535] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 7992.640538] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7998.042398] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7998.264511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8003.408953] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8003.687687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8008.170607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8008.175823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8008.177466] Lustre: Skipped 4 previous similar messages [ 8008.183938] Lustre: Skipped 7 previous similar messages [ 8011.961225] Lustre: server umount lustre-OST0000 complete [ 8016.866312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8016.869894] LustreError: Skipped 1 previous similar message [ 8030.177630] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8030.505057] Lustre: server umount lustre-MDT0000 complete [ 8033.760517] LustreError: 218530:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8033.798923] LustreError: 218530:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8034.846750] LustreError: 219314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483703 with bad export cookie 1389986970491376644 [ 8034.853946] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8034.863371] LustreError: 219314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8035.289594] Lustre: server umount lustre-MDT0001 complete [ 8044.199093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8044.737688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8044.751079] Lustre: Skipped 4 previous similar messages [ 8048.944970] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8056.469970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8061.676214] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8069.398971] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8075.798928] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8082.572685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8089.420259] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8096.554923] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8106.467068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8108.631403] Lustre: server umount lustre-OST0000 complete [ 8118.338673] Lustre: server umount lustre-MDT0000 complete [ 8122.203435] LustreError: 222689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483791 with bad export cookie 1389986970491377533 [ 8122.217651] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8122.777246] Lustre: server umount lustre-MDT0001 complete [ 8237.407353] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 8240.868648] Key type lgssc unregistered [ 8241.152586] LNet: 226370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8241.159230] LNetError: 226370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8241.183582] LNet: Removed LNI 192.168.201.148@tcp [ 8242.204712] Key type .llcrypt unregistered [ 8242.208975] Key type ._llcrypt unregistered [ 8268.377606] Key type ._llcrypt registered [ 8268.384309] Key type .llcrypt registered [ 8268.563165] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 8284.091626] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 8285.468604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8285.557800] alg: No test for adler32 (adler32-zlib) [ 8286.836345] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 8287.133920] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 8288.912926] Key type lgssc registered [ 8290.752423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8325.294015] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8325.347483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8326.561861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8326.586484] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8326.647637] Lustre: lustre-MDT0000: new disk, initializing [ 8326.708542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8326.721307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8330.723751] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8340.703572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8340.793503] Lustre: 229722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8340.878593] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8340.884326] Lustre: Skipped 1 previous similar message [ 8340.983453] Lustre: lustre-MDT0001: new disk, initializing [ 8341.094914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8341.221208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8341.235836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8345.429345] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8354.505377] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8359.941781] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8367.083374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8367.358607] Lustre: lustre-OST0000: new disk, initializing [ 8367.362077] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8367.367342] Lustre: 231690:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8367.459071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8373.283195] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8376.995812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8377.010955] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8377.238335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8382.378879] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8386.815426] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8387.026752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8391.578948] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8391.812781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8397.794039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8397.798106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8397.804126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8397.810299] Lustre: Skipped 1 previous similar message [ 8400.060670] Lustre: server umount lustre-OST0000 complete [ 8407.524570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8407.536621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8407.551070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8407.559186] Lustre: Skipped 1 previous similar message [ 8410.013513] Lustre: server umount lustre-MDT0000 complete [ 8413.636942] LustreError: 229715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484082 with bad export cookie 1725090740417837568 [ 8413.640036] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8413.660261] LustreError: 229715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8414.048607] Lustre: server umount lustre-MDT0001 complete [ 8430.252421] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 8439.216993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8439.633764] LustreError: 234623:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8439.645996] LustreError: 234623:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8439.684113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8443.772207] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8444.911887] LustreError: 234624:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8450.017215] LustreError: 234623:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8452.147559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8452.432068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8456.431775] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8459.218655] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8465.863259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8466.224433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8472.075349] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8481.535948] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8481.665526] Lustre: lustre-OST0001: new disk, initializing [ 8481.668396] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8481.673976] Lustre: 236930:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8488.570313] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8490.563454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8490.577930] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8490.670577] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8499.246881] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8504.731970] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8516.695792] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 10:56:24 (1789484184) [ 8519.138404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8519.150070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8519.175419] Lustre: Skipped 1 previous similar message [ 8519.185400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8519.192412] Lustre: Skipped 2 previous similar messages [ 8521.702233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8521.709432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8521.726900] Lustre: Skipped 2 previous similar messages [ 8525.065509] Lustre: server umount lustre-MDT0000 complete [ 8526.822878] LustreError: 234623:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8526.842042] LustreError: 234623:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8528.867764] LustreError: 234603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484197 with bad export cookie 1725090740417838443 [ 8528.876872] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8528.879795] LustreError: 234603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8529.229821] Lustre: server umount lustre-MDT0001 complete [ 8542.237788] Lustre: server umount lustre-OST0000 complete [ 8557.635443] Lustre: server umount lustre-OST0001 complete [ 8564.513254] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 8577.065957] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 8615.500920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8615.801610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8615.847763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8615.947392] Lustre: lustre-MDT0000: new disk, initializing [ 8616.075613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8616.086731] Lustre: Skipped 1 previous similar message [ 8616.111343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8620.748384] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8632.444578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8632.584499] Lustre: 240873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8632.639272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8632.643540] Lustre: Skipped 1 previous similar message [ 8632.739774] Lustre: lustre-MDT0001: new disk, initializing [ 8632.940905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8632.947598] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8638.757742] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8649.298957] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8655.617781] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8664.784040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8665.319058] Lustre: lustre-OST0000: new disk, initializing [ 8665.327676] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8665.340523] Lustre: 242845:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8665.464134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8665.481113] Lustre: Skipped 1 previous similar message [ 8667.148652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8667.181432] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8667.336429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8673.720289] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8684.297561] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8688.312266] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8688.561313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8693.963743] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8694.236318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8698.338126] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8698.341933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8698.373074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8698.409240] Lustre: Skipped 4 previous similar messages [ 8702.571107] Lustre: server umount lustre-OST0000 complete [ 8708.580673] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8708.587650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8708.607599] Lustre: Skipped 2 previous similar messages [ 8708.622343] Lustre: Skipped 1 previous similar message [ 8716.027955] Lustre: server umount lustre-MDT0000 complete [ 8718.822541] LustreError: 242458:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8718.846911] LustreError: 242458:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8720.140910] LustreError: 240868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484389 with bad export cookie 1725090740417840585 [ 8720.161026] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8720.168341] LustreError: 240868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8720.937293] Lustre: server umount lustre-MDT0001 complete [ 8733.560541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8734.373518] LustreError: 245065:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8734.472634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8739.714449] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8748.640432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8754.308176] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8763.528371] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8770.451686] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8778.405989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8784.877887] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 8792.034542] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8943.074341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8943.075575] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8943.081772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8943.081781] Lustre: Skipped 2 previous similar messages [ 8943.108702] Lustre: Skipped 1 previous similar message [ 8953.311133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8953.622667] Lustre: server umount lustre-OST0000 complete [ 8963.865244] Lustre: server umount lustre-MDT0000 complete [ 8967.879788] LustreError: 245047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484636 with bad export cookie 1725090740417841467 [ 8967.882623] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8967.894075] LustreError: 245047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8968.315393] Lustre: server umount lustre-MDT0001 complete [ 9063.376974] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 9066.552294] Key type lgssc unregistered [ 9066.903648] LNet: 248684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9066.914736] LNetError: 248684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9066.936709] LNet: Removed LNI 192.168.201.148@tcp [ 9067.684277] Key type .llcrypt unregistered [ 9067.686478] Key type ._llcrypt unregistered [ 9094.679976] Key type ._llcrypt registered [ 9094.681838] Key type .llcrypt registered [ 9094.772279] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 9112.124871] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 9113.716216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9113.925207] alg: No test for adler32 (adler32-zlib) [ 9115.330067] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 9115.525419] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 9117.184336] Key type lgssc registered [ 9118.714828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9172.472446] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 9184.844404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9184.868565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9186.107178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9186.128746] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9186.220984] Lustre: lustre-MDT0000: new disk, initializing [ 9186.336764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9186.365922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9191.297917] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9200.997514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9201.145367] Lustre: 252752:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9201.202653] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9201.206345] Lustre: Skipped 1 previous similar message [ 9201.278389] Lustre: lustre-MDT0001: new disk, initializing [ 9201.410369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9201.469560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9201.482661] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9206.853528] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9211.978743] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9218.787848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9219.075109] Lustre: lustre-OST0000: new disk, initializing [ 9219.089730] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9219.096075] Lustre: 254383:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9219.242794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9225.335300] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9227.820578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9227.840816] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9227.946397] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9235.199652] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9235.401765] Lustre: lustre-OST0001: new disk, initializing [ 9235.404702] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9235.410307] Lustre: 255254:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9235.560032] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9241.720891] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9241.724377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9241.743677] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9241.860592] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9254.232672] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9259.592990] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9272.801604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9272.810454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9272.828539] Lustre: Skipped 3 previous similar messages [ 9272.851708] Lustre: Skipped 3 previous similar messages [ 9277.255343] Lustre: server umount lustre-MDT0000 complete [ 9280.323432] LustreError: 252742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484949 with bad export cookie 16130180609561079588 [ 9280.328936] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9280.333290] LustreError: 252742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9280.623455] Lustre: server umount lustre-MDT0001 complete [ 9294.485491] Lustre: server umount lustre-OST0000 complete [ 9308.842099] Lustre: server umount lustre-OST0001 complete [ 9322.543450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9332.694859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9341.822434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9353.298134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9378.380798] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 9391.361237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9391.405040] Lustre: lustre-MDT0000: reset Object Index mappings [ 9392.221819] LustreError: 260270:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9392.251917] LustreError: 260270:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9392.381020] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9397.732047] LustreError: 260271:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9398.325822] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9402.853965] LustreError: 260270:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9407.970837] LustreError: 260271:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9408.151207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9408.165420] Lustre: lustre-MDT0001: reset Object Index mappings [ 9408.493112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9414.159715] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9417.717177] Lustre: 261413:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9425.720037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9426.078455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9431.895696] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9433.260968] LustreError: 262203:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9437.372681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9441.124568] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9446.909872] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9448.711382] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9456.998899] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9461.565513] Lustre: 263283:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9468.005647] Lustre: server umount lustre-MDT0000 complete [ 9469.925897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9469.934132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9469.948883] LustreError: 260270:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9469.967038] LustreError: 260270:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9472.084744] LustreError: 261790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789485141 with bad export cookie 16130180609561086378 [ 9472.093813] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9472.097262] LustreError: 261790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9472.394897] Lustre: server umount lustre-MDT0001 complete [ 9489.375387] Lustre: 250257:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789485141/real 1789485141] req@ffff8aaa35919f80 x1876410796994176/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789485157 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9489.409440] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9493.472562] Lustre: 250256:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789485146/real 1789485146] req@ffff8aa9086e3480 x1876410796994816/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789485162 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9493.516290] Lustre: 250256:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9495.794908] Lustre: server umount lustre-OST0000 complete [ 9498.079208] Lustre: 250254:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789485151/real 1789485151] req@ffff8aa9089b4a80 x1876410796995200/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789485167 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9498.117551] Lustre: 250254:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9500.178776] Lustre: server umount lustre-OST0001 complete [ 9521.318357] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:13:09 (1789485189) [ 9547.274508] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 9560.829452] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 9604.410875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9604.988346] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9605.016247] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9605.081751] Lustre: lustre-MDT0000: new disk, initializing [ 9605.134093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9605.142508] Lustre: Skipped 1 previous similar message [ 9605.156658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9609.726449] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9622.600064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9622.761558] Lustre: 267539:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9622.871909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9622.878560] Lustre: Skipped 1 previous similar message [ 9623.023920] Lustre: lustre-MDT0001: new disk, initializing [ 9623.259166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9623.286675] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9628.806125] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9640.533856] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9647.552860] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9658.003284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9658.370409] Lustre: lustre-OST0000: new disk, initializing [ 9658.375570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9658.386804] Lustre: 269513:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9659.727639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9659.745151] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9659.957918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9666.582587] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9678.090656] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9683.285304] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9683.477940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9687.546342] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9687.770633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9690.594092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9690.603464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9690.635550] Lustre: Skipped 1 previous similar message [ 9690.647630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9690.658564] Lustre: Skipped 1 previous similar message [ 9695.734105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9695.742111] Lustre: Skipped 1 previous similar message [ 9696.344617] Lustre: server umount lustre-OST0000 complete [ 9700.834440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9700.835952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9700.843771] Lustre: Skipped 2 previous similar messages [ 9705.976900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9705.987917] Lustre: Skipped 2 previous similar messages [ 9707.079480] Lustre: server umount lustre-MDT0000 complete [ 9711.077808] LustreError: 267551:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9711.107563] LustreError: 267551:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9712.823074] LustreError: 268335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789485381 with bad export cookie 16130180609561088471 [ 9712.832410] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9712.840799] LustreError: 268335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9713.562914] Lustre: server umount lustre-MDT0001 complete [ 9736.270994] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 9748.093871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9748.575593] LustreError: 272448:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9748.639093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9748.644190] Lustre: Skipped 2 previous similar messages [ 9753.283361] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9764.340626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9770.355149] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9773.558968] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9780.869260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9787.968749] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9797.085227] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9797.285824] Lustre: lustre-OST0001: new disk, initializing [ 9797.290880] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9797.299185] Lustre: 274758:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9799.146205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9799.162466] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9799.317477] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9805.534598] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9814.728480] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9818.537596] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9823.618541] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9823.630794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9834.981925] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9834.991546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9835.002565] Lustre: Skipped 2 previous similar messages [ 9835.033519] Lustre: Skipped 3 previous similar messages [ 9835.854098] Lustre: server umount lustre-MDT0000 complete [ 9840.098171] LustreError: 272428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789485509 with bad export cookie 16130180609561089353 [ 9840.113261] LustreError: 272444:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9840.119961] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9840.127639] LustreError: 272428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9840.202361] LustreError: 272444:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9840.702237] Lustre: server umount lustre-MDT0001 complete [ 9855.747532] Lustre: server umount lustre-OST0000 complete [ 9870.904823] Lustre: server umount lustre-OST0001 complete [ 9890.932387] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 9901.460672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9901.802624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9901.816437] Lustre: Skipped 3 previous similar messages [ 9907.470484] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9916.484738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9923.615949] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9927.778307] Lustre: 279261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9936.291553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9945.147374] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9946.103656] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9957.506111] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9963.066201] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9967.802873] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 9977.799735] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9981.805598] Lustre: 281139:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9987.361303] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9987.484471] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10002.223289] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:21:09 (1789485669) [10019.297240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10019.313322] LustreError: Skipped 1 previous similar message [10019.315982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10019.322818] Lustre: Skipped 1 previous similar message [10019.329109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10023.883236] Lustre: server umount lustre-MDT0000 complete [10027.493776] LustreError: 279433:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10027.510888] LustreError: 279433:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [10028.822474] LustreError: 278102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789485697 with bad export cookie 16130180609561095744 [10028.843053] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10028.848488] LustreError: 278102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10043.359240] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10044.011987] Lustre: server umount lustre-MDT0001 complete [10054.908774] Lustre: server umount lustre-OST0000 complete [10059.537320] Lustre: server umount lustre-OST0001 complete [10067.850448] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [10079.426735] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [10136.353580] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [10148.000952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10148.212658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10148.235871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10148.314280] Lustre: lustre-MDT0000: new disk, initializing [10148.395389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10153.697762] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10168.031334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10168.138834] Lustre: 285680:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10168.398582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10168.405280] Lustre: Skipped 4 previous similar messages [10168.442090] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10173.379782] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10185.572129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10185.884516] Lustre: 287312:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10187.223341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10193.912491] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10207.734514] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10208.003990] Lustre: 288185:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10209.665414] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10209.689423] Lustre: Skipped 1 previous similar message [10209.890244] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10219.329964] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10231.373835] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10235.758730] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10235.764612] Lustre: Skipped 1 previous similar message [10255.848996] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 11:25:23 (1789485923) [10258.919378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10258.921615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10258.932832] Lustre: Skipped 8 previous similar messages [10258.950032] Lustre: Skipped 13 previous similar messages [10260.595725] Lustre: server umount lustre-MDT0000 complete [10264.820258] LustreError: 285670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789485933 with bad export cookie 16130180609561103668 [10264.826827] LustreError: 285670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10264.832215] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10265.238275] Lustre: server umount lustre-MDT0001 complete [10280.116831] Lustre: server umount lustre-OST0000 complete [10294.440992] Lustre: server umount lustre-OST0001 complete [10304.109911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10304.800989] LustreError: 290436:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10304.825729] LustreError: 290436:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [10310.599933] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10319.516237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10325.624301] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10335.694700] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10343.397919] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10350.810345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10358.638821] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10368.163365] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10397.567470] loop1: detected capacity change from 0 to 204800000 [10397.818421] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10397.831558] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10397.849153] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10404.251814] loop1: detected capacity change from 0 to 204800000 [10410.781936] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10410.953780] LustreError: 290426:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs5332: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10410.968287] LustreError: 290426:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs5332 [10410.972798] LustreError: 290426:0:(mgs_handler.c:517:mgs_target_reg()) fs5332-MDT0001: Error checking target index: rc = -13 [10410.990370] LustreError: fs5332-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10411.001282] LustreError: 294157:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10411.010552] LustreError: 294157:0:(tgt_mount.c:2128:server_put_super()) no obd fs5332-MDT0001 [10411.022586] LustreError: 294157:0:(tgt_mount.c:116:server_deregister_mount()) fs5332-MDT0001 not registered [10411.078232] Lustre: server umount fs5332-MDT0001 complete [10411.081195] LustreError: 294157:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10421.315335] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10421.585450] Lustre: Setting parameter fs5332-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs5332-MDT0001 [10421.687036] Lustre: srv-fs5332-MDT0001: No data found on store. Initialize space. [10421.693105] Lustre: Skipped 4 previous similar messages [10421.859088] Lustre: fs5332-MDT0001: new disk, initializing [10421.864740] Lustre: Skipped 3 previous similar messages [10422.058332] Lustre: srv-fs5332-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10422.066446] Lustre: 294623:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5332-MDT0001-mdtlov: can't create new llog: rc = -115 [10422.082111] LustreError: 294623:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5332-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10428.789553] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10436.532599] Lustre: server umount fs5332-MDT0001 complete [10445.806068] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10450.932215] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10451.836303] loop1: detected capacity change from 0 to 204800000 [10457.886498] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10458.255591] Lustre: 296193:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5332-MDT0001-mdtlov: can't create new llog: rc = -115 [10458.270328] LustreError: 296193:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5332-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10463.356552] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10468.644393] Lustre: server umount fs5332-MDT0001 complete [10481.634966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10481.637648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10481.652028] Lustre: Skipped 1 previous similar message [10481.658143] Lustre: Skipped 2 previous similar messages [10485.127927] Lustre: server umount lustre-MDT0000 complete [10494.920435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10495.112603] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10499.789606] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10500.604251] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10508.210464] loop1: detected capacity change from 0 to 204800000 [10513.534544] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10514.002394] Lustre: 298640:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5332-MDT0001-mdtlov: can't create new llog: rc = -115 [10514.025893] LustreError: 298640:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5332-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10519.482801] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10525.252252] Lustre: server umount fs5332-MDT0001 complete [10549.218800] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10555.444450] Lustre: server umount lustre-OST0000 complete [10566.040410] Lustre: server umount lustre-MDT0000 complete [10570.258201] LustreError: 290417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789486239 with bad export cookie 16130180609561106349 [10570.263407] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10570.266087] LustreError: 290417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10570.721701] Lustre: server umount lustre-MDT0001 complete [10581.991583] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [10585.199119] Key type lgssc unregistered [10585.506952] LNet: 300985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10585.513551] LNetError: 300985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10586.539942] LNet: Removed LNI 192.168.201.148@tcp [10587.492145] Key type .llcrypt unregistered [10587.494536] Key type ._llcrypt unregistered [10608.678660] Key type ._llcrypt registered [10608.684555] Key type .llcrypt registered [10608.841722] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [10609.722488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10609.786234] alg: No test for adler32 (adler32-zlib) [10611.018711] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [10611.300380] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [10613.007229] Key type lgssc registered [10614.303269] Lustre: Echo OBD driver; http://www.lustre.org/ [10622.153504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10622.170799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10623.783696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10628.758602] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10633.205890] LustreError: 302252:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10633.217931] LustreError: 302252:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10638.307263] LustreError: 302251:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10638.519187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10638.869156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10644.589170] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10653.344267] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10659.413275] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10666.626317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10667.063857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10668.073544] LustreError: 304080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10673.133030] LustreError: 304080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10674.699902] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10678.256857] LustreError: 304153:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10682.289588] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10688.505349] LustreError: 304153:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10688.515688] LustreError: 304153:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10701.726813] loop1: detected capacity change from 0 to 204800000 [10701.964084] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10701.980994] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10701.989102] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10705.807132] LustreError: 304153:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10705.838051] LustreError: 304153:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [10708.631636] loop1: detected capacity change from 0 to 204800000 [10714.670724] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10714.889421] Lustre: fs2231-OST0000: new disk, initializing [10714.894849] Lustre: srv-fs2231-OST0000: No data found on store. Initialize space. [10714.910241] Lustre: 305992:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10715.034628] Lustre: fs2231-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10721.516343] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10729.982801] Lustre: server umount fs2231-OST0000 complete [10739.687375] LustreError: 304080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10739.724613] LustreError: 304080:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [10742.817926] loop1: detected capacity change from 0 to 204800000 [10748.871781] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10749.175753] Lustre: fs2231-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10749.933963] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10755.252500] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10765.298458] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10765.396462] loop2: detected capacity change from 0 to 204800000 [10765.802333] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10765.831127] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10765.843475] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10771.563734] loop2: detected capacity change from 0 to 204800000 [10778.317559] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10778.412129] LustreError: fs2231-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10778.429922] LustreError: fs2231-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10778.440790] LustreError: 309034:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10778.444753] LustreError: 309034:0:(tgt_mount.c:2128:server_put_super()) no obd fs2231-OST0001 [10778.452125] LustreError: 309034:0:(tgt_mount.c:116:server_deregister_mount()) fs2231-OST0001 not registered [10778.474790] Lustre: server umount fs2231-OST0001 complete [10778.478099] LustreError: 309034:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10791.806420] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10792.079638] Lustre: fs2231-OST0001: new disk, initializing [10792.087464] Lustre: srv-fs2231-OST0001: No data found on store. Initialize space. [10792.107464] Lustre: 309585:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10792.282573] Lustre: fs2231-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10800.315637] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10806.266394] LustreError: 304082:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10806.299164] LustreError: 304082:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [10809.786306] Lustre: server umount fs2231-OST0000 complete [10819.935276] Lustre: server umount fs2231-OST0001 complete [10831.675343] loop1: detected capacity change from 0 to 204800000 [10837.306938] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10837.546679] Lustre: fs2231-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10844.909169] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10852.816024] loop2: detected capacity change from 0 to 204800000 [10858.379484] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10865.660383] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [10875.903245] Lustre: server umount fs2231-OST0000 complete [10888.491313] Lustre: server umount fs2231-OST0001 complete [10903.530641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10903.531772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10903.542941] LustreError: Skipped 2 previous similar messages [10903.545475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10903.561662] Lustre: Skipped 1 previous similar message [10906.962877] Lustre: server umount lustre-OST0000 complete [10913.760764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10913.768845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10913.782064] Lustre: Skipped 1 previous similar message [10913.777485] Lustre: Skipped 1 previous similar message [10918.425870] Lustre: server umount lustre-MDT0000 complete [10922.344287] LustreError: 302232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789486591 with bad export cookie 7622229950101823609 [10922.356144] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10922.359251] LustreError: 302232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10922.760381] Lustre: server umount lustre-MDT0001 complete [10933.635185] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [10936.365282] Key type lgssc unregistered [10936.663190] LNet: 315509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10936.672760] LNetError: 315509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10936.705209] LNet: Removed LNI 192.168.201.148@tcp [10937.489157] Key type .llcrypt unregistered [10937.493311] Key type ._llcrypt unregistered [10968.682378] Key type ._llcrypt registered [10968.685330] Key type .llcrypt registered [10968.781569] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [10984.562244] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [10985.787556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10985.878863] alg: No test for adler32 (adler32-zlib) [10986.912739] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [10987.146618] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [10988.833359] Key type lgssc registered [10990.010688] Lustre: Echo OBD driver; http://www.lustre.org/ [11029.655984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11029.676905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11031.068296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11031.151418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11031.262179] Lustre: lustre-MDT0000: new disk, initializing [11031.358378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11031.388939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11037.385193] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11049.032660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11049.152386] Lustre: 319368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11049.254932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11049.261163] Lustre: Skipped 1 previous similar message [11049.350650] Lustre: lustre-MDT0001: new disk, initializing [11049.426861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11049.449413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11049.456033] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11053.717650] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11063.409561] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11069.116569] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11075.881151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11076.154884] Lustre: lustre-OST0000: new disk, initializing [11076.158322] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11076.165516] Lustre: 321338:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11076.234668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11082.314200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11082.327268] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11082.483155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11082.682934] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11092.774142] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11094.945818] Lustre: Mounted lustre-client - version 2.17.58_40_gc6c2479 [11095.975360] Lustre: Unmounted lustre-client [11105.253843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11105.268076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11105.278828] Lustre: Skipped 1 previous similar message [11110.393080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11110.399757] Lustre: Skipped 1 previous similar message [11111.479367] Lustre: server umount lustre-OST0000 complete [11115.501671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11115.509864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11115.518872] Lustre: Skipped 2 previous similar messages [11115.535300] Lustre: Skipped 1 previous similar message [11120.610167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11120.620479] Lustre: Skipped 1 previous similar message [11121.629395] Lustre: server umount lustre-MDT0000 complete [11125.373945] LustreError: 319361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789486794 with bad export cookie 563454844091551525 [11125.395559] LustreError: 319361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11125.399183] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11125.869291] Lustre: server umount lustre-MDT0001 complete [11137.188308] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [11139.842144] Key type lgssc unregistered [11156.527126] LNet: 323502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11156.531366] LNetError: 323502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11224.166323] LNet: Removed LNI 192.168.201.148@tcp [11225.193583] Key type .llcrypt unregistered [11225.197604] Key type ._llcrypt unregistered [11244.419122] Key type ._llcrypt registered [11244.421727] Key type .llcrypt registered [11244.529933] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [11262.021756] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [11263.625026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11263.701051] alg: No test for adler32 (adler32-zlib) [11264.806858] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [11265.114660] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [11266.895290] Key type lgssc registered [11268.416080] Lustre: Echo OBD driver; http://www.lustre.org/ [11299.444342] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 11:42:47 (1789486967) [11306.455338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11306.499296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11307.754795] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11307.816531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11307.967959] Lustre: lustre-MDT0000: new disk, initializing [11308.038708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11308.065951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11312.844248] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11324.914800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11325.092082] Lustre: 326723:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11325.204694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11325.214218] Lustre: Skipped 1 previous similar message [11325.327386] Lustre: lustre-MDT0001: new disk, initializing [11325.423503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11325.458270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11325.470066] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11331.036343] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11342.169458] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11350.485536] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11358.508834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11358.823160] Lustre: lustre-OST0000: new disk, initializing [11358.830567] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11358.843393] Lustre: 328693:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11358.955433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11365.032580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11365.055015] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11365.183294] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11365.999364] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11376.122581] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11396.071765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11396.072116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11396.124516] LustreError: Skipped 1 previous similar message [11396.134203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11398.867669] Lustre: server umount lustre-OST0000 complete [11406.304572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11406.311882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11406.316522] Lustre: Skipped 2 previous similar messages [11406.330741] Lustre: Skipped 1 previous similar message [11409.844499] Lustre: server umount lustre-MDT0000 complete [11414.185326] LustreError: 326714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789487083 with bad export cookie 16818714084073266334 [11414.189940] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11414.195356] LustreError: 326714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11414.585023] Lustre: server umount lustre-MDT0001 complete [11426.325861] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [11429.009518] Key type lgssc unregistered [11429.483858] LNet: 330601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11429.509711] LNetError: 330601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11429.531816] LNet: Removed LNI 192.168.201.148@tcp [11430.485152] Key type .llcrypt unregistered [11430.488356] Key type ._llcrypt unregistered [11452.282920] Key type ._llcrypt registered [11452.284496] Key type .llcrypt registered [11452.393385] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [11454.001604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11454.096749] alg: No test for adler32 (adler32-zlib) [11455.215229] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [11455.438034] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [11457.079215] Key type lgssc registered [11458.331062] Lustre: Echo OBD driver; http://www.lustre.org/ [11467.074575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11467.093805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11468.494974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11472.838845] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11479.035275] LustreError: 331871:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11479.054244] LustreError: 331871:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11481.496110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11481.968954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11486.184678] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11494.016124] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11499.444511] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11506.502876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11506.672511] LustreError: 333692:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11506.928681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11514.136315] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11522.539421] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11531.222804] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11532.628441] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11538.400970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11538.404459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11538.416712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11539.851321] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.48@tcp (stopping) [11539.943475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11539.968087] Lustre: Skipped 1 previous similar message [11543.413382] Lustre: server umount lustre-MDT0000 complete [11544.938196] LustreError: 331867:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11548.094023] LustreError: 332606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789487217 with bad export cookie 9619161654973606335 [11548.106309] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11548.107203] LustreError: 332606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11550.068400] LustreError: 331868:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11550.094155] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.48@tcp (stopping) [11550.114905] LustreError: 331868:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11550.142966] Lustre: Skipped 3 previous similar messages [11550.182666] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11553.195412] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11553.208827] Lustre: Skipped 1 previous similar message [11554.869597] Lustre: server umount lustre-MDT0001 complete [11559.122623] Lustre: server umount lustre-OST0000 complete [11569.208344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11573.765129] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11576.380181] Lustre: 336049:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.148@tcp [11578.484463] Lustre: server umount MGS complete [11588.284776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11588.713037] LustreError: 336621:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11588.728026] LustreError: 336621:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11588.811696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11593.876484] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11598.306612] LustreError: 336622:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11598.333459] LustreError: 336622:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11602.743345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11602.994218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11608.370623] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11617.314259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11617.569254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11624.434434] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11633.547593] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11635.687423] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11642.778519] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11644.646848] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11651.052353] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11651.067454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11651.079243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11652.027279] Lustre: server umount lustre-OST0000 complete [11659.233815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11659.258453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11659.285564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11662.252728] Lustre: server umount lustre-MDT0000 complete [11664.869689] LustreError: 336622:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11664.881852] LustreError: 336622:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11665.392540] LustreError: 336603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789487334 with bad export cookie 9619161654973607175 [11665.395818] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11665.405908] LustreError: 336603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11675.823784] Lustre: server umount lustre-MDT0001 complete [11688.379451] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [11691.871616] Key type lgssc unregistered [11692.339763] LNet: 340058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11692.345454] LNetError: 340058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11692.374239] LNet: Removed LNI 192.168.201.148@tcp [11693.390239] Key type .llcrypt unregistered [11693.393708] Key type ._llcrypt unregistered [11717.129858] Key type ._llcrypt registered [11717.135983] Key type .llcrypt registered [11717.324966] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [11737.537686] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [11738.423071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11738.546463] alg: No test for adler32 (adler32-zlib) [11739.636356] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [11739.893304] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [11741.546038] Key type lgssc registered [11743.066653] Lustre: Echo OBD driver; http://www.lustre.org/ [11786.645366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11786.720789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11788.167294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11788.209672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11788.290775] Lustre: lustre-MDT0000: new disk, initializing [11788.433089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11788.460520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11793.922971] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11804.657809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11804.850185] Lustre: 343222:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11804.916581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11804.921849] Lustre: Skipped 1 previous similar message [11804.995814] Lustre: lustre-MDT0001: new disk, initializing [11805.080877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11805.132237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11805.146448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11809.058197] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11818.217046] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11824.136047] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11831.036988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11831.384259] Lustre: lustre-OST0000: new disk, initializing [11831.391984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11831.398232] Lustre: 345191:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11831.486102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11838.221829] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11840.574886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11840.584398] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11840.710085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11849.460484] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11855.669374] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11855.957912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11859.986660] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11860.261314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11866.598223] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11866.599184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11866.602156] LustreError: Skipped 1 previous similar message [11866.605400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11866.628979] Lustre: Skipped 1 previous similar message [11868.874507] Lustre: server umount lustre-OST0000 complete [11876.837430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11876.842904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11876.854135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11876.884174] Lustre: Skipped 2 previous similar messages [11879.960847] Lustre: server umount lustre-MDT0000 complete [11884.592924] LustreError: 343213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789487553 with bad export cookie 5440756823744990700 [11884.610894] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11884.613662] LustreError: 343213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11884.940869] Lustre: server umount lustre-MDT0001 complete [11897.275662] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 11:52:44 (1789487564) [11915.519046] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [11926.426160] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [11964.455023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11964.700362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11964.726937] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11964.818499] Lustre: lustre-MDT0000: new disk, initializing [11964.890529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11964.905023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11969.163827] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11979.957896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11980.124389] Lustre: 349954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11980.189021] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11980.196368] Lustre: Skipped 1 previous similar message [11980.284376] Lustre: lustre-MDT0001: new disk, initializing [11980.364546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11980.413792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11980.430088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11985.224172] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [11995.277570] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12000.339970] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12006.770943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12007.058967] Lustre: lustre-OST0000: new disk, initializing [12007.062428] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12007.068691] Lustre: 351919:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12007.139827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12013.230272] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12014.143893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12014.151420] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12014.368786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12022.680702] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12055.522715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12055.533940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12055.549208] Lustre: Skipped 1 previous similar message [12055.564664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12055.564814] LustreError: Skipped 1 previous similar message [12055.595104] Lustre: Skipped 1 previous similar message [12060.641573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12060.659425] Lustre: Skipped 1 previous similar message [12065.761513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12065.776663] Lustre: Skipped 1 previous similar message [12068.321244] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12068.561387] Lustre: server umount lustre-OST0000 complete [12072.423771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12072.435232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12072.445571] Lustre: Skipped 1 previous similar message [12076.002918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12076.012636] Lustre: Skipped 2 previous similar messages [12078.476256] Lustre: server umount lustre-MDT0000 complete [12079.009146] LustreError: 349964:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12079.017543] LustreError: 349964:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12082.004581] LustreError: 350757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789487751 with bad export cookie 5440756823744991582 [12082.010056] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12082.014186] LustreError: 350757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12082.310252] Lustre: server umount lustre-MDT0001 complete [12092.624336] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12095.255615] Key type lgssc unregistered [12095.507813] LNet: 354092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12095.517523] LNetError: 354092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12095.534088] LNet: Removed LNI 192.168.201.148@tcp [12096.320302] Key type .llcrypt unregistered [12096.321907] Key type ._llcrypt unregistered [12190.418805] Key type ._llcrypt registered [12190.425910] Key type .llcrypt registered [12190.537361] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12190.909485] Key type .llcrypt unregistered [12190.912684] Key type ._llcrypt unregistered [12266.358651] Key type ._llcrypt registered [12266.360730] Key type .llcrypt registered [12266.581618] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12266.855301] Key type .llcrypt unregistered [12266.857309] Key type ._llcrypt unregistered [12296.885788] Key type ._llcrypt registered [12296.888580] Key type .llcrypt registered [12296.995793] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12297.357312] Key type .llcrypt unregistered [12297.362095] Key type ._llcrypt unregistered [12326.745962] Key type ._llcrypt registered [12326.749947] Key type .llcrypt registered [12326.848722] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12327.131371] Key type .llcrypt unregistered [12327.134565] Key type ._llcrypt unregistered [12402.096308] Key type ._llcrypt registered [12402.104210] Key type .llcrypt registered [12402.192435] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12402.538489] Key type .llcrypt unregistered [12402.541870] Key type ._llcrypt unregistered [12434.883813] Key type ._llcrypt registered [12434.885937] Key type .llcrypt registered [12435.052589] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12435.401848] Key type .llcrypt unregistered [12435.404908] Key type ._llcrypt unregistered [12467.614554] Key type ._llcrypt registered [12467.616885] Key type .llcrypt registered [12467.724937] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12467.997336] Key type .llcrypt unregistered [12467.999059] Key type ._llcrypt unregistered [12503.032860] Key type ._llcrypt registered [12503.034460] Key type .llcrypt registered [12503.104526] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12503.396282] Key type .llcrypt unregistered [12503.399494] Key type ._llcrypt unregistered [12536.916874] Key type ._llcrypt registered [12536.919686] Key type .llcrypt registered [12537.068801] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12537.357355] Key type .llcrypt unregistered [12537.360198] Key type ._llcrypt unregistered [12571.765595] Key type ._llcrypt registered [12571.772640] Key type .llcrypt registered [12571.904565] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12572.166603] Key type .llcrypt unregistered [12572.168902] Key type ._llcrypt unregistered [12638.112379] Key type ._llcrypt registered [12638.119887] Key type .llcrypt registered [12638.289202] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12638.728245] Key type .llcrypt unregistered [12638.734733] Key type ._llcrypt unregistered [12674.116244] Key type ._llcrypt registered [12674.118847] Key type .llcrypt registered [12674.176208] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12674.499278] Key type .llcrypt unregistered [12674.502364] Key type ._llcrypt unregistered [12709.656825] Key type ._llcrypt registered [12709.658519] Key type .llcrypt registered [12709.755760] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [12709.986416] Key type .llcrypt unregistered [12709.990108] Key type ._llcrypt unregistered [12766.857937] Key type ._llcrypt registered [12766.862919] Key type .llcrypt registered [12766.968067] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [12783.770683] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [12784.988890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12785.135420] alg: No test for adler32 (adler32-zlib) [12786.349592] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [12786.666211] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [12788.304109] Key type lgssc registered [12789.492913] Lustre: Echo OBD driver; http://www.lustre.org/ [12825.098915] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12825.153697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12826.480787] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12826.542519] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12826.653458] Lustre: tst250fs-MDT0000: new disk, initializing [12826.793591] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12826.825916] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12831.701771] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12844.495840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12844.710769] Lustre: 374400:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12844.802368] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12844.806515] Lustre: Skipped 1 previous similar message [12844.852554] Lustre: tst250fs-MDT0001: new disk, initializing [12844.997532] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12845.064694] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12845.080249] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12849.727859] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12854.796205] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12862.020686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12862.462152] Lustre: tst250fs-OST0000: new disk, initializing [12862.465961] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12862.471488] Lustre: 376033:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12862.608269] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12869.203541] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12869.213849] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12869.233597] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12869.410883] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12881.821686] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12882.029487] Lustre: tst250fs-OST0001: new disk, initializing [12882.035990] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12882.048139] Lustre: 376904:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12882.191492] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12888.218935] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [12891.798895] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12891.815703] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12891.955917] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12893.928976] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12896.328020] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12896.333481] Lustre: Skipped 2 previous similar messages [12916.706103] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12916.717274] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12916.735304] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12917.729078] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12917.747128] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12917.769916] Lustre: Skipped 2 previous similar messages [12921.203977] Lustre: server umount tst250fs-MDT0000 complete [12926.312524] LustreError: 375662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789488595 with bad export cookie 11914877400833192049 [12926.339531] LustreError: 375662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12926.342549] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12926.706661] Lustre: server umount tst250fs-MDT0001 complete [12942.035328] Lustre: server umount tst250fs-OST0000 complete [12943.329585] Lustre: 372622:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789488596/real 1789488596] req@ffff8aaa40df2d80 x1876414646257408/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789488612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12943.383657] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12946.093760] Lustre: server umount tst250fs-OST0001 complete [12966.071426] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [12975.005685] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [13010.756857] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:11:18 (1789488678) [13012.564572] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13028.921872] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [13041.251867] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [13073.679402] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12774 sec ========= 12:12:21 (1789488741) [13075.588107] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:12:23 (1789488743) === [13079.237717] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:12:27 (1789488747) === [13124.778592] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [13127.677897] Key type lgssc unregistered [13127.944643] LNet: 386033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13127.953538] LNetError: 386033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13127.968657] LNet: Removed LNI 192.168.201.148@tcp [13128.989178] Key type .llcrypt unregistered [13128.992278] Key type ._llcrypt unregistered