[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 484361795 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002012] Switched APIC routing to physical x2apic. [ 0.003020] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008014] pid_max: default: 32768 minimum: 301 [ 0.009130] LSM: Security Framework initializing [ 0.010048] Yama: becoming mindful. [ 0.011030] SELinux: Initializing. [ 0.012066] *** VALIDATE selinux *** [ 0.020744] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025570] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028089] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030031] *** VALIDATE tmpfs *** [ 0.031429] *** VALIDATE proc *** [ 0.033186] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.036066] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037143] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039037] Spectre V2 : User space: Vulnerable [ 0.040000] Speculative Store Bypass: Vulnerable [ 0.042441] debug: unmapping init [mem 0xffffffffb5459000-0xffffffffb5460fff] [ 0.044155] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045682] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046021] ... version: 2 [ 0.047010] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050012] ... max period: 00007fffffffffff [ 0.051013] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.053296] rcu: Hierarchical SRCU implementation. [ 0.055414] smp: Bringing up secondary CPUs ... [ 0.056580] x86: Booting SMP configuration: [ 0.057020] .... node #0, CPUs: #1 #2 #3 [ 0.063667] smp: Brought up 1 node, 4 CPUs [ 0.065016] smpboot: Max logical packages: 1 [ 0.066012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.119000] node 0 deferred pages initialised in 49ms [ 0.123193] devtmpfs: initialized [ 0.124312] x86/mm: Memory block size: 128MB [ 0.128047] gcov: version magic: 0x41383552 [ 0.131359] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.135148] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.138434] pinctrl core: initialized pinctrl subsystem [ 0.140167] [ 0.140836] ************************************************************* [ 0.142016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.145014] ** ** [ 0.149022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.152014] ** ** [ 0.154013] ** This means that this kernel is built to expose internal ** [ 0.157016] ** IOMMU data structures, which may compromise security on ** [ 0.160014] ** your system. ** [ 0.164051] ** ** [ 0.167020] ** If you see this message and you are not debugging the ** [ 0.172033] ** kernel, report this immediately to your vendor! ** [ 0.175167] ** ** [ 0.177115] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.179012] ************************************************************* [ 0.184881] NET: Registered protocol family 16 [ 0.187968] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.191127] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.193088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.197065] cpuidle: using governor menu [ 0.199000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205553] PCI: Using configuration type 1 for base access [ 0.208168] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.217116] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.218051] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.220095] cryptd: max_cpu_qlen set to 1000 [ 0.222213] ACPI: Added _OSI(Module Device) [ 0.223029] ACPI: Added _OSI(Processor Device) [ 0.224000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.225011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.228213] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.234648] ACPI: Interpreter enabled [ 0.235040] ACPI: PM: (supports S0 S3 S4 S5) [ 0.236000] ACPI: Using IOAPIC for interrupt routing [ 0.237097] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.240361] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.249833] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.251028] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.254016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.256069] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.260128] acpiphp: Slot [2] registered [ 0.262129] acpiphp: Slot [3] registered [ 0.263064] acpiphp: Slot [4] registered [ 0.264060] acpiphp: Slot [5] registered [ 0.265118] acpiphp: Slot [6] registered [ 0.266077] acpiphp: Slot [7] registered [ 0.267068] acpiphp: Slot [8] registered [ 0.269076] acpiphp: Slot [9] registered [ 0.270089] acpiphp: Slot [10] registered [ 0.271156] acpiphp: Slot [11] registered [ 0.273071] acpiphp: Slot [12] registered [ 0.275058] acpiphp: Slot [13] registered [ 0.276132] acpiphp: Slot [14] registered [ 0.277067] acpiphp: Slot [15] registered [ 0.280084] acpiphp: Slot [16] registered [ 0.283116] acpiphp: Slot [17] registered [ 0.286093] acpiphp: Slot [18] registered [ 0.290123] acpiphp: Slot [19] registered [ 0.291084] acpiphp: Slot [20] registered [ 0.293076] acpiphp: Slot [21] registered [ 0.296229] acpiphp: Slot [22] registered [ 0.297070] acpiphp: Slot [23] registered [ 0.300200] acpiphp: Slot [24] registered [ 0.301108] acpiphp: Slot [25] registered [ 0.302127] acpiphp: Slot [26] registered [ 0.303071] acpiphp: Slot [27] registered [ 0.304065] acpiphp: Slot [28] registered [ 0.305067] acpiphp: Slot [29] registered [ 0.306100] acpiphp: Slot [30] registered [ 0.307087] acpiphp: Slot [31] registered [ 0.308113] PCI host bridge to bus 0000:00 [ 0.310040] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.312034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.315037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.316030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.318026] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.320030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.321255] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.325000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.329979] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.339013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.343053] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.346031] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.349020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.351017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.354425] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.356814] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.359043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.362609] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.367012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.379000] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.384020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.389000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.397050] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.403040] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.416026] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.429983] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.436024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.453025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.466024] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.480724] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.490023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.510022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.536023] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.558200] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.577109] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.593026] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.619024] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.630850] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.646036] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.654023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.679024] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.698000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.707031] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.713029] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.758027] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.780733] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.783389] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.785233] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.787568] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.789266] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.797055] iommu: Default domain type: Passthrough [ 0.802604] SCSI subsystem initialized [ 0.804184] ACPI: bus type USB registered [ 0.806162] usbcore: registered new interface driver usbfs [ 0.808150] usbcore: registered new interface driver hub [ 0.810126] usbcore: registered new device driver usb [ 0.812192] pps_core: LinuxPPS API ver. 1 registered [ 0.813012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.816110] PTP clock support registered [ 0.817161] EDAC MC: Ver: 3.0.0 [ 0.820539] PCI: Using ACPI for IRQ routing [ 0.823000] NetLabel: Initializing [ 0.823000] NetLabel: domain hash size = 128 [ 0.823018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.826114] NetLabel: unlabeled traffic allowed by default [ 0.829554] vgaarb: loaded [ 0.832661] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.835026] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.845104] clocksource: Switched to clocksource kvm-clock [ 0.966307] VFS: Disk quotas dquot_6.6.0 [ 0.967876] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.970639] *** VALIDATE ramfs *** [ 0.971844] *** VALIDATE hugetlbfs *** [ 0.973699] pnp: PnP ACPI init [ 0.976404] pnp: PnP ACPI: found 6 devices [ 0.998132] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.001452] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.003392] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.005552] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.007203] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.008964] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 1.011317] NET: Registered protocol family 2 [ 1.013818] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.018177] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.021453] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.026599] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.030371] TCP: Hash tables configured (established 65536 bind 65536) [ 1.033219] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.036343] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.039371] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.042516] NET: Registered protocol family 1 [ 1.047267] RPC: Registered named UNIX socket transport module. [ 1.050172] RPC: Registered udp transport module. [ 1.052354] RPC: Registered tcp transport module. [ 1.054351] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.056874] NET: Registered protocol family 44 [ 1.058539] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.060469] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.062741] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.065074] PCI: CLS 0 bytes, default 64 [ 1.066838] Unpacking initramfs... [ 3.025704] debug: unmapping init [mem 0xffff8d347cc54000-0xffff8d347ffbffff] [ 3.029986] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.032274] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.035246] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.715586] Initialise system trusted keyrings [ 3.717143] Key type blacklist registered [ 3.720077] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.729283] zbud: loaded [ 3.734767] *** VALIDATE nfs *** [ 3.738040] *** VALIDATE nfs4 *** [ 3.741808] pstore: using deflate compression [ 3.757219] Platform Keyring initialized [ 3.971580] NET: Registered protocol family 38 [ 3.972928] Key type asymmetric registered [ 3.974166] Asymmetric key parser 'x509' registered [ 3.975806] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.978506] io scheduler mq-deadline registered [ 3.980104] io scheduler kyber registered [ 3.981904] io scheduler bfq registered [ 3.984923] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.988939] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.992370] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.994548] ACPI: Power Button [PWRF] [ 4.095253] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.209390] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.543992] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.746531] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.086525] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.115743] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.147143] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.155942] Non-volatile memory driver v1.3 [ 5.157571] Linux agpgart interface v0.103 [ 5.201532] virtio_blk virtio1: [vda] 133912 512-byte logical blocks (68.6 MB/65.4 MiB) [ 5.204666] vda: detected capacity change from 0 to 68562944 [ 5.223959] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.228432] vdb: detected capacity change from 0 to 1073741824 [ 5.256105] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.258822] vdc: detected capacity change from 0 to 2621440000 [ 5.300303] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.305012] vdd: detected capacity change from 0 to 2621440000 [ 5.330173] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.336163] vde: detected capacity change from 0 to 4294967296 [ 5.363866] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.368412] vdf: detected capacity change from 0 to 4294967296 [ 5.380290] libphy: Fixed MDIO Bus: probed [ 5.398719] usbcore: registered new interface driver usbserial_generic [ 5.407694] usbserial: USB Serial support registered for generic [ 5.409735] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.417620] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.419349] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.426344] mousedev: PS/2 mouse device common for all mice [ 5.429192] rtc_cmos 00:05: RTC can wake from S4 [ 5.432526] rtc_cmos 00:05: registered as rtc0 [ 5.437992] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.439499] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.447552] intel_pstate: CPU model not supported [ 5.461806] hid: raw HID events driver (C) Jiri Kosina [ 5.464888] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.482708] usbcore: registered new interface driver usbhid [ 5.482719] usbhid: USB HID core driver [ 5.482852] drop_monitor: Initializing network drop monitor service [ 5.482989] Initializing XFRM netlink socket [ 5.483404] NET: Registered protocol family 10 [ 5.484773] Segment Routing with IPv6 [ 5.484808] NET: Registered protocol family 17 [ 5.489684] mpls_gso: MPLS GSO support [ 5.506113] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.521138] RAS: Correctable Errors collector initialized. [ 5.522905] AVX version of gcm_enc/dec engaged. [ 5.524217] AES CTR mode by8 optimization enabled [ 5.707072] sched_clock: Marking stable (5707051846, 0)->(6669737022, -962685176) [ 5.721695] registered taskstats version 1 [ 5.727539] Loading compiled-in X.509 certificates [ 5.731726] zswap: loaded using pool lzo/zbud [ 5.802506] Key type big_key registered [ 5.837184] Key type encrypted registered [ 5.843411] ima: No TPM chip found, activating TPM-bypass! [ 5.849407] ima: Allocated hash algorithm: sha1 [ 5.850880] ima: No architecture policies found [ 5.853271] evm: Initialising EVM extended attributes: [ 5.855995] evm: security.selinux [ 5.858498] evm: security.ima [ 5.860481] evm: security.capability [ 5.862739] evm: HMAC attrs: 0x1 [ 5.871987] rtc_cmos 00:05: setting system clock to 2025-10-24 05:19:30 UTC (1761283170) [ 5.882808] debug: unmapping init [mem 0xffffffffb6403000-0xffffffffb65fffff] [ 5.887632] debug: unmapping init [mem 0xffffffffb5182000-0xffffffffb5458fff] [ 5.892167] Write protecting the kernel read-only data: 28672k [ 5.897048] debug: unmapping init [mem 0xffffffffb3803000-0xffffffffb39fffff] [ 5.905537] debug: unmapping init [mem 0xffffffffb4114000-0xffffffffb41fffff] [ 5.989290] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.008506] systemd[1]: Detected virtualization kvm. [ 6.012452] systemd[1]: Detected architecture x86-64. [ 6.015465] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.069667] systemd[1]: No hostname configured. [ 6.071181] systemd[1]: Set hostname to . [ 6.072903] random: systemd: uninitialized urandom read (16 bytes read) [ 6.082477] systemd[1]: Initializing machine ID from random generator. [ 6.215432] random: ln: uninitialized urandom read (6 bytes read) [ 6.559981] random: systemd: uninitialized urandom read (16 bytes read) [ 6.570367] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 6.585772] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ 6.611622] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.785990] device-mapper: uevent: version 1.0.3 [ 7.788155] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.460547] random: fast init done Starting dracut initqueue hook... [ 9.549733] virtio_net virtio0 ens2: renamed from eth0 [ 10.096638] scsi host0: ata_piix [ 10.253758] scsi host1: ata_piix [ 10.268154] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.270028] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.384783] random: crng init done [ 14.389615] random: 7 urandom warning(s) missed due to ratelimiting [ 16.239793] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.243285] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.067667] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.688604] SELinux: Disabled at runtime. [ 19.808097] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.828797] systemd[1]: Detected virtualization kvm. [ 19.835695] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.020786] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.031482] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.066353] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.070215] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.072463] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.090180] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.118080] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ 21.571298] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 21.977089] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.695620] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.778690] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.034351] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.121910] EDAC sbridge: Ver: 1.1.2 [ 27.514693] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (6s / no limit)[ 28.291902] NFS: Registering the id_resolver key type [ 28.295457] Key type id_resolver registered [ 28.298343] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg238-server login: [ 56.056457] libcfs: loading out-of-tree module taints kernel. [ 56.070279] Key type ._llcrypt registered [ 56.071721] Key type .llcrypt registered [ 56.119396] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 64.014422] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 64.609901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 64.617662] alg: No test for adler32 (adler32-zlib) [ 65.708381] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 66.051357] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 67.671161] Key type lgssc registered [ 68.340181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 75.296199] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 75.747599] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 79.027424] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 81.886792] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 85.007980] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 91.185360] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 96.447519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 96.484723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 96.496938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 97.599230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 97.614819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 97.654033] Lustre: lustre-MDT0000: new disk, initializing [ 97.678526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 97.685973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 99.095578] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 105.374574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 105.402819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 105.432112] Lustre: 6488:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 105.444270] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 105.446793] Lustre: Skipped 1 previous similar message [ 105.487579] Lustre: lustre-MDT0001: new disk, initializing [ 105.516590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 105.527236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 105.530957] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 105.543063] LustreError: 7442:0:(llog_cat.c:511:llog_cat_current_log()) lustre-MDT0001-osd: next log does not exist, catalog [0x240000400:0x1:0x0] rc=-5 [ 105.548048] LustreError: 7442:0:(update_trans.c:965:top_trans_stop()) lustre-MDT0001-osd: write updates failed: rc = -5 [ 105.552284] LustreError: 7442:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 107.051253] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 109.612789] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 113.282040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 113.304582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 113.390328] Lustre: lustre-OST0000: new disk, initializing [ 113.391742] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 113.411053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 115.435277] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 116.140361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 116.144153] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 116.167580] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 121.132428] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 121.157925] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 121.194547] Lustre: lustre-OST0001: new disk, initializing [ 121.196364] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 121.218065] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 121.524314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 121.528156] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 121.543509] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 123.148158] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 129.253858] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 136.304264] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 143.596524] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing check_logdir /tmp/testlogs/ [ 145.314513] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing yml_node [ 146.974403] Lustre: DEBUG MARKER: Client: 2.16.59.47 [ 148.003919] Lustre: DEBUG MARKER: MDS: 2.16.59.47 [ 148.971968] Lustre: DEBUG MARKER: OSS: 2.16.59.47 [ 149.529869] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 24 01:21:53 EDT 2025 [ 156.142812] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 156.722029] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 172.000117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 172.002903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 172.008225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 173.025376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 173.026140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 173.034262] Lustre: Skipped 3 previous similar messages [ 178.144093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 178.147268] Lustre: Skipped 3 previous similar messages [ 183.263209] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 183.264441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 183.270131] Lustre: Skipped 3 previous similar messages [ 183.316741] LustreError: 12193:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 183.360400] Lustre: server umount lustre-MDT0000 complete [ 184.969037] LustreError: 6480:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761283349 with bad export cookie 3006483125567924353 [ 184.971101] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 184.973880] LustreError: 6480:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 188.383935] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 188.384502] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 188.388680] Lustre: Skipped 3 previous similar messages [ 191.104743] LustreError: 12397:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 191.107699] LustreError: 12397:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 191.188027] Lustre: server umount lustre-MDT0001 complete [ 192.796140] LustreError: 12598:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 192.798789] LustreError: 12598:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 192.825569] Lustre: server umount lustre-OST0000 complete [ 194.609163] Lustre: server umount lustre-OST0001 complete [ 197.406158] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 201.409199] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 205.908657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 208.921088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 211.642168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 214.880874] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 218.241336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 218.264052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 218.368563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 218.384268] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 218.425159] Lustre: lustre-MDT0000: new disk, initializing [ 218.465544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 218.474254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 220.326110] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 225.564361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 225.591298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 225.634579] Lustre: 14978:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 225.767930] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 225.813757] Lustre: 15777:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 225.818538] Lustre: 15777:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 225.822265] Lustre: 15777:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 225.828135] Lustre: 15777:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 225.831915] Lustre: 15777:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 225.834868] Lustre: 15777:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 227.260945] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 230.601365] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 231.954394] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 234.363324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 234.388810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 234.495757] Lustre: lustre-OST0000: new disk, initializing [ 234.497287] Lustre: Skipped 1 previous similar message [ 234.499715] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 234.502855] Lustre: Skipped 2 previous similar messages [ 234.532094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 234.534344] Lustre: Skipped 1 previous similar message [ 235.579495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 235.583024] Lustre: Skipped 1 previous similar message [ 235.584677] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 235.617032] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 236.648972] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 240.158920] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 241.733105] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 241.822755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 243.346140] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 243.438226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 245.732460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 245.735188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 245.736928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 245.740201] Lustre: Skipped 1 previous similar message [ 245.743336] Lustre: Skipped 3 previous similar messages [ 250.286082] LustreError: 17917:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 250.288689] LustreError: 17917:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 250.356042] Lustre: server umount lustre-OST0000 complete [ 255.969911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 255.973811] Lustre: Skipped 1 previous similar message [ 266.207202] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 266.208902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 266.212967] Lustre: Skipped 6 previous similar messages [ 266.251175] LustreError: 18118:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 266.253320] LustreError: 18118:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 266.289914] Lustre: server umount lustre-MDT0000 complete [ 267.811694] LustreError: 15760:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761283432 with bad export cookie 3006483125567925956 [ 267.815348] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 267.816310] LustreError: 15760:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 267.987300] Lustre: server umount lustre-MDT0001 complete [ 270.549329] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:23:54 (1761283434) [ 277.835301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 278.035192] LustreError: 19456:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 278.043499] LustreError: 19456:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [ 278.067105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 279.699116] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 283.104650] LustreError: 19456:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 283.471368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 285.449312] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 287.793169] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 289.257516] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 290.191283] Lustre: Failing over lustre-MDT0000 [ 290.261984] LustreError: 20799:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 290.264423] LustreError: 20799:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 290.308702] Lustre: server umount lustre-MDT0000 complete [ 292.834240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 292.835627] LustreError: 19455:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 292.839523] Lustre: Skipped 1 previous similar message [ 292.848017] LustreError: 19455:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 298.976339] LustreError: 19455:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.983081] LustreError: 19455:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 304.096101] LustreError: 19455:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 304.102857] LustreError: 19455:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 304.128837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 304.177607] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 305.770166] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 308.386489] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 308.390296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 308.398994] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 309.365313] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 311.556402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 311.561987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 311.564583] Lustre: Skipped 1 previous similar message [ 317.696688] LustreError: 21778:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 317.699466] LustreError: 21778:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 317.735923] Lustre: server umount lustre-MDT0000 complete [ 319.279626] LustreError: 19481:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761283483 with bad export cookie 3006483125567927412 [ 319.282554] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 319.285331] LustreError: 19481:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 319.459221] Lustre: server umount lustre-MDT0001 complete [ 323.295305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 323.500522] LustreError: 22457:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.533428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 323.536643] Lustre: Skipped 2 previous similar messages [ 325.392255] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 329.379031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 331.308227] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 333.650823] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 335.281549] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 336.266174] Lustre: Failing over lustre-MDT0000 [ 336.396644] Lustre: server umount lustre-MDT0000 complete [ 339.936395] LustreError: 22456:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 339.942147] LustreError: 22456:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 4 previous similar messages [ 350.445801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 350.522476] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 352.331409] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 354.720600] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 354.724127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 354.726486] Lustre: Skipped 1 previous similar message [ 354.734430] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 356.086231] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 358.335111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 358.343200] Lustre: Skipped 4 previous similar messages [ 364.484815] LustreError: 24781:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 364.487975] LustreError: 24781:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 364.555601] Lustre: server umount lustre-MDT0000 complete [ 366.158464] LustreError: 22438:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761283530 with bad export cookie 3006483125567928343 [ 366.162786] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 366.163791] LustreError: 22438:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 366.324428] Lustre: server umount lustre-MDT0001 complete [ 370.075147] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:25:34 (1761283534) [ 376.674644] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 381.311612] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 385.783680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 388.723077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 391.369480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 393.943702] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 397.698036] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 401.665992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 401.690772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 401.779615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 401.792686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 401.834894] Lustre: lustre-MDT0000: new disk, initializing [ 401.864898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 401.867497] Lustre: Skipped 2 previous similar messages [ 401.874707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 403.366062] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 407.953173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 407.983759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 408.025927] Lustre: 28445:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 408.156572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 408.204838] Lustre: 29244:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 408.208690] Lustre: 29244:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 408.214468] Lustre: 29244:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 408.220871] Lustre: 29244:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 408.227251] Lustre: 29244:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 408.231362] Lustre: 29244:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 408.235849] Lustre: 29244:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 408.239637] Lustre: 29244:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 408.243353] Lustre: 29244:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 408.247064] Lustre: 29244:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 408.250819] Lustre: 29244:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 408.254290] Lustre: 29244:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 409.613692] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 412.913946] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 414.261714] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 416.733458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 416.761225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 418.883883] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 418.912539] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 418.919304] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 418.925756] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 422.105900] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 422.397095] Lustre: 28452:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 511, rollback = 2 [ 422.400487] Lustre: 28452:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 422.403576] Lustre: 28452:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 2/8/6, destroy: 0/0/0 [ 422.407103] Lustre: 28452:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 422.410352] Lustre: 28452:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 5/511/0 [ 422.412967] Lustre: 28452:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 422.415787] Lustre: 28452:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 5/58/0, punch: 0/0/0, quota 1/3/0 [ 422.419099] Lustre: 28452:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 422.421944] Lustre: 28452:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 5/102/3, delete: 0/0/0 [ 422.424822] Lustre: 28452:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 422.427894] Lustre: 28452:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 422.431250] Lustre: 28452:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 424.956123] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 434.145989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 434.146433] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 434.150844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 434.153449] LustreError: Skipped 1 previous similar message [ 434.155940] Lustre: Skipped 6 previous similar messages [ 436.419119] LustreError: 31083:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 436.422131] LustreError: 31083:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 436.502039] Lustre: server umount lustre-OST0000 complete [ 438.751942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 444.057534] Lustre: server umount lustre-MDT0000 complete [ 444.384073] LustreError: 29232:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 444.390224] LustreError: 29232:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [ 445.464735] LustreError: 28437:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761283610 with bad export cookie 3006483125567928742 [ 445.469241] LustreError: 28437:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 445.469786] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 445.699670] Lustre: server umount lustre-MDT0001 complete [ 449.914727] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 451.125541] Key type lgssc unregistered [ 451.268649] LNet: 32016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 451.272573] LNetError: 32016:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 451.279457] LNet: Removed LNI 192.168.202.138@tcp [ 451.656142] Key type .llcrypt unregistered [ 451.657595] Key type ._llcrypt unregistered [ 462.527407] Key type ._llcrypt registered [ 462.528803] Key type .llcrypt registered [ 462.577070] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 463.034712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 463.049325] alg: No test for adler32 (adler32-zlib) [ 463.917525] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 464.010743] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 465.599170] Key type lgssc registered [ 466.028415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 469.155838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 469.166776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 470.381469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 471.866723] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 475.197837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 475.346682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 476.759600] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 478.793894] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 480.109691] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 482.536478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 482.669812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 484.663304] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 484.711896] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 486.532455] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 501.706481] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:27:46 (1761283666) [ 502.356697] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 502.977555] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 505.071345] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:27:49 (1761283669) [ 507.848050] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:27:52 (1761283672) [ 508.179854] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 511.025378] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 511.027980] Lustre: Skipped 2 previous similar messages [ 511.601978] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:27:55 (1761283675) [ 515.107249] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 515.110027] Lustre: Skipped 7 previous similar messages [ 519.541289] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:28:03 (1761283683) [ 520.463367] Lustre: *** cfs_fail_loc=131b, val=2*** [ 521.609528] Lustre: *** cfs_fail_loc=131b, val=2*** [ 524.335396] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:28:08 (1761283688) [ 524.975789] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 524.979184] Lustre: Skipped 6 previous similar messages [ 529.220869] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:28:13 (1761283693) [ 536.034673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 536.035648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.042298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 536.044641] Lustre: Skipped 1 previous similar message [ 539.244091] LustreError: 39939:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 539.302623] Lustre: server umount lustre-OST0000 complete [ 541.153538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 541.154422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 541.157575] Lustre: Skipped 2 previous similar messages [ 546.272548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 546.274774] Lustre: Skipped 2 previous similar messages [ 546.623977] LustreError: 40142:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 546.626352] LustreError: 40142:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 546.656560] Lustre: server umount lustre-MDT0000 complete [ 547.910102] LustreError: 35996:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761283712 with bad export cookie 16636750253797588163 [ 547.913086] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 547.913504] LustreError: 35996:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 547.972845] LustreError: 40343:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 547.975589] LustreError: 40343:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 548.052745] Lustre: server umount lustre-MDT0001 complete [ 551.609076] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 552.607383] Key type lgssc unregistered [ 552.735410] LNet: 40874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 552.738470] LNetError: 40874:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 552.751442] LNet: Removed LNI 192.168.202.138@tcp [ 553.053546] Key type .llcrypt unregistered [ 553.054965] Key type ._llcrypt unregistered [ 558.752094] Key type ._llcrypt registered [ 558.753450] Key type .llcrypt registered [ 558.791745] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 565.159492] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 565.469434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 565.500072] alg: No test for adler32 (adler32-zlib) [ 566.335825] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 566.410654] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 567.991135] Key type lgssc registered [ 568.373092] Lustre: Echo OBD driver; http://www.lustre.org/ [ 571.271257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 573.289931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 575.214780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 577.220341] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 579.458903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 579.482547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 579.488649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 580.572806] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 580.583569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 580.620072] Lustre: lustre-MDT0000: new disk, initializing [ 580.642048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 580.648853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 581.890605] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 585.952679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 585.972040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 586.003235] Lustre: 43974:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 586.024917] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 586.027736] Lustre: Skipped 1 previous similar message [ 586.060707] Lustre: lustre-MDT0001: new disk, initializing [ 586.099165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 586.111985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 586.116131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 586.155939] Lustre: 44773:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 586.161039] Lustre: 44773:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 586.164852] Lustre: 44773:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 586.169654] Lustre: 44773:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 586.173669] Lustre: 44773:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 586.176496] Lustre: 44773:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 587.471224] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 590.727442] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 591.914666] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 594.249671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 594.273077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 594.389710] Lustre: lustre-OST0000: new disk, initializing [ 594.392491] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 594.430442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 596.450543] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 599.538312] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 600.887520] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 604.157741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 604.161960] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 604.192932] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 605.037283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 606.396421] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 606.470826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 609.251616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 609.252621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 609.253370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 609.254855] Lustre: Skipped 1 previous similar message [ 609.256769] LustreError: Skipped 1 previous similar message [ 613.166842] LustreError: 46942:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 613.226334] Lustre: server umount lustre-OST0000 complete [ 616.928170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 616.931859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 616.937811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 616.940171] Lustre: Skipped 1 previous similar message [ 619.488911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 619.489074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.547460] LustreError: 47143:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 620.549563] LustreError: 47143:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 620.580471] Lustre: server umount lustre-MDT0000 complete [ 621.759193] LustreError: 43965:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761283786 with bad export cookie 3963829935330974972 [ 621.761425] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 621.762994] LustreError: 43965:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 621.828168] LustreError: 47344:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 621.830069] LustreError: 47344:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 621.907357] Lustre: server umount lustre-MDT0001 complete [ 625.027868] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:29:49 (1761283789) [ 627.254296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 627.432497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 628.747386] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 631.604393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 631.718599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 632.999318] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 634.700387] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 635.880580] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 638.205045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 640.290115] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 642.190579] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 654.498830] LustreError: 50279:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 654.783831] Lustre: Modifying parameter general.timeout=1 in log params [ 655.990809] Lustre: Modifying parameter general.timeout=5 in log params [ 655.992954] Lustre: Skipped 3 previous similar messages [ 658.069661] Lustre: Modifying parameter general.timeout=12 in log params [ 658.071733] Lustre: Skipped 6 previous similar messages [ 662.355106] Lustre: Modifying parameter general.timeout=26 in log params [ 662.357846] Lustre: Skipped 13 previous similar messages [ 670.417535] Lustre: Modifying parameter general.timeout=53 in log params [ 670.419106] Lustre: Skipped 26 previous similar messages [ 686.551794] Lustre: Modifying parameter general.timeout=108 in log params [ 686.554161] Lustre: Skipped 54 previous similar messages [ 696.420400] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:31:00 (1761283860) [ 697.391827] Lustre: 57178:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 698.596807] Lustre: 57374:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 698.601264] Lustre: 57374:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 1 previous similar message [ 709.234445] Lustre: 59032:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 709.237711] Lustre: 59032:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 3 previous similar messages [ 709.239567] LustreError: 59032:0:(mgs_handler.c:1126:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 710.525316] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:31:14 (1761283874) [ 716.257042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 716.258321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 716.260274] Lustre: Skipped 2 previous similar messages [ 716.263521] Lustre: Skipped 3 previous similar messages [ 720.314222] LustreError: 59730:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 720.316464] LustreError: 59730:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 720.341806] Lustre: server umount lustre-MDT0000 complete [ 721.377342] LustreError: 48061:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 721.381882] LustreError: 48061:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 4 previous similar messages [ 721.497221] LustreError: 48041:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761283886 with bad export cookie 3963829935330975868 [ 721.500357] LustreError: 48041:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 721.501270] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 721.637801] Lustre: server umount lustre-MDT0001 complete [ 733.040842] LustreError: 60131:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 733.042747] LustreError: 60131:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 733.122305] Lustre: server umount lustre-OST0000 complete [ 735.919348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 737.198321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 738.403101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 739.637343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 741.825111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 741.907659] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 741.917319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 741.920637] Lustre: Skipped 44 previous similar messages [ 742.003541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 742.006972] Lustre: Skipped 1 previous similar message [ 743.204366] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 745.843975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 745.873456] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 745.883286] Lustre: 61577:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 747.391328] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 749.271736] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 750.418811] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 752.606831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 752.670858] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 754.478160] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 756.145158] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 756.659937] Lustre: 63810:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 763.364639] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 763.365419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 763.367608] LustreError: Skipped 1 previous similar message [ 763.373767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 763.375846] Lustre: Skipped 1 previous similar message [ 764.461708] LustreError: 64057:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 764.464594] LustreError: 64057:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 764.530317] Lustre: server umount lustre-OST0000 complete [ 766.431774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 773.599977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 773.602692] Lustre: Skipped 4 previous similar messages [ 780.255110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 780.317697] Lustre: server umount lustre-MDT0000 complete [ 781.551191] LustreError: 61570:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761283946 with bad export cookie 3963829935330978178 [ 781.553570] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 781.555084] LustreError: 61570:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 781.612143] LustreError: 64469:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 781.613961] LustreError: 64469:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 781.690826] Lustre: server umount lustre-MDT0001 complete [ 785.174763] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 786.133829] Key type lgssc unregistered [ 786.245349] LNet: 65000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.248841] LNetError: 65000:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.256304] LNet: Removed LNI 192.168.202.138@tcp [ 786.563042] Key type .llcrypt unregistered [ 786.563945] Key type ._llcrypt unregistered [ 795.059522] Key type ._llcrypt registered [ 795.060681] Key type .llcrypt registered [ 795.100666] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 795.580218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 795.601034] alg: No test for adler32 (adler32-zlib) [ 796.440651] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 796.521380] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 798.103191] Key type lgssc registered [ 798.447852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 801.017082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 801.024759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 802.194494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 803.531031] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 806.508049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 806.714485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 808.050800] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 809.868573] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 811.131255] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 813.544725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 813.729483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 815.699246] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 817.622789] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 830.979725] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 837.604918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 837.606064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 837.606366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 837.608144] Lustre: Skipped 1 previous similar message [ 839.416257] LustreError: 69099:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 839.437729] Lustre: server umount lustre-MDT0000 complete [ 840.597145] LustreError: 66350:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761284005 with bad export cookie 1021884318839672863 [ 840.599666] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 840.602901] LustreError: 66350:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 840.658114] LustreError: 69303:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 840.659759] LustreError: 69303:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 840.729828] Lustre: server umount lustre-MDT0001 complete [ 852.212213] LustreError: 69503:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 852.214413] LustreError: 69503:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 852.283678] Lustre: server umount lustre-OST0000 complete [ 855.363500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 856.654863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 857.908572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 859.172187] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 861.450027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 861.567263] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 861.577903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 861.662135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 862.991465] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 865.875646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 865.912470] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 865.923874] Lustre: 70952:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 866.097051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 867.440685] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 869.203827] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 870.370616] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 872.638059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 872.705346] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 874.805679] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 876.602521] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 877.126873] Lustre: 73195:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 877.734620] Lustre: Modifying parameter general.jobid_var=disable in log params [ 883.168661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 883.169176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 883.171110] LustreError: Skipped 1 previous similar message [ 883.174103] Lustre: Skipped 1 previous similar message [ 883.174788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 883.178480] Lustre: Skipped 3 previous similar messages [ 885.038131] LustreError: 73446:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 885.040107] LustreError: 73446:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 885.115544] Lustre: server umount lustre-OST0000 complete [ 886.751726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 886.753720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 886.758625] Lustre: Skipped 1 previous similar message [ 886.761073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 892.474349] LustreError: 73653:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 892.476035] LustreError: 73653:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 892.504454] Lustre: server umount lustre-MDT0000 complete [ 893.408785] LustreError: 70962:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 893.412975] LustreError: 70962:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 893.693327] LustreError: 71683:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761284058 with bad export cookie 1021884318839673675 [ 893.695920] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 893.696517] LustreError: 71683:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 893.822191] Lustre: server umount lustre-MDT0001 complete [ 897.234108] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 898.299210] Key type lgssc unregistered [ 898.420388] LNet: 74384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.423347] LNetError: 74384:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.430312] LNet: Removed LNI 192.168.202.138@tcp [ 898.722936] Key type .llcrypt unregistered [ 898.724051] Key type ._llcrypt unregistered [ 908.044601] Key type ._llcrypt registered [ 908.045921] Key type .llcrypt registered [ 908.086666] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 908.412293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 908.438247] alg: No test for adler32 (adler32-zlib) [ 909.288233] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 909.371188] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 910.951141] Key type lgssc registered [ 911.307230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 913.934996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 913.942413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 915.122296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 916.471195] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 919.523389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 919.757344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 921.122891] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 922.976136] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 924.211804] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 926.563633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 926.761506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 928.769686] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 930.677874] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 943.648069] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 944.270657] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 944.273306] Lustre: Skipped 1 previous similar message [ 945.546268] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 945.549642] Lustre: Skipped 3 previous similar messages [ 947.690438] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 947.692682] Lustre: Skipped 6 previous similar messages [ 951.739387] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 951.742442] Lustre: Skipped 12 previous similar messages [ 959.785719] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 959.787789] Lustre: Skipped 24 previous similar messages [ 978.113035] Lustre: 83510:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 978.441384] Lustre: Disabling parameter general.jobid_var= in log params [ 978.443300] Lustre: Skipped 48 previous similar messages [ 979.029293] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 01:35:43 (1761284143) [ 979.596039] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 980.205425] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 01:35:44 (1761284144) [ 986.083480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 986.083765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 986.085692] LustreError: Skipped 1 previous similar message [ 986.086431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 986.090209] Lustre: Skipped 1 previous similar message [ 987.310133] LustreError: 83803:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 987.390374] Lustre: server umount lustre-OST0000 complete [ 991.201031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 991.201181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 991.201560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 991.201565] Lustre: Skipped 1 previous similar message [ 991.208860] Lustre: Skipped 1 previous similar message [ 994.748801] LustreError: 84007:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 994.751144] LustreError: 84007:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 994.777207] Lustre: server umount lustre-MDT0000 complete [ 996.056531] LustreError: 77385:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761284160 with bad export cookie 1958502426884570337 [ 996.060613] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 996.061693] LustreError: 77385:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 996.132288] LustreError: 84210:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 996.135012] LustreError: 84210:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 996.233168] Lustre: server umount lustre-MDT0001 complete [ 999.946207] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1001.012380] Key type lgssc unregistered [ 1001.149394] LNet: 84741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.151989] LNetError: 84741:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1001.161360] LNet: Removed LNI 192.168.202.138@tcp [ 1001.473757] Key type .llcrypt unregistered [ 1001.475106] Key type ._llcrypt unregistered [ 1003.014954] Key type ._llcrypt registered [ 1003.016356] Key type .llcrypt registered [ 1003.060500] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1004.535985] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules [ 1005.006248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1005.020698] alg: No test for adler32 (adler32-zlib) [ 1005.885106] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 1005.972997] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1007.559157] Key type lgssc registered [ 1007.998522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1008.419173] LustreError: 85857:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 1016.112693] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1019.283112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1029.544515] LustreError: 85857:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1029.552872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1030.751820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1032.151148] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1034.860263] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 01:36:39 (1761284199) [ 1036.375268] LustreError: 86895:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1036.378464] LustreError: 86895:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1036.380852] LustreError: 86895:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1036.441713] LustreError: 87668:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1036.519380] Lustre: server umount lustre-MDT0000 complete [ 1040.719808] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 1041.696764] Key type lgssc unregistered [ 1041.823407] LNet: 88355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.826157] LNetError: 88355:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1041.837275] LNet: Removed LNI 192.168.202.138@tcp [ 1042.128355] Key type .llcrypt unregistered [ 1042.129424] Key type ._llcrypt unregistered [ 1048.779822] Key type ._llcrypt registered [ 1048.781126] Key type .llcrypt registered [ 1048.818921] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1049.192743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1049.257220] alg: No test for adler32 (adler32-zlib) [ 1050.111284] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 1050.200762] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 1051.791188] Key type lgssc registered [ 1052.134820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1054.640581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1054.646059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1055.796090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1057.054536] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1059.791087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1059.999654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1061.284181] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1063.033414] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1064.173066] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1066.370716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1066.574796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1068.379149] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1070.106833] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1091.823114] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:37:36 (1761284256) [ 1092.461919] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1093.104204] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:37:37 (1761284257) [ 1095.649252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1095.650034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1095.651590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1095.658870] Lustre: Skipped 2 previous similar messages [ 1100.220282] LustreError: 92439:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1100.245175] Lustre: server umount lustre-MDT0000 complete [ 1100.769910] LustreError: 90246:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1100.775299] LustreError: 90246:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 4 previous similar messages [ 1101.456472] LustreError: 89511:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761284266 with bad export cookie 4236484533288371638 [ 1101.460602] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1101.462365] LustreError: 89511:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1101.533887] LustreError: 92640:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1101.535502] LustreError: 92640:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1101.627837] Lustre: server umount lustre-MDT0001 complete [ 1113.139522] LustreError: 92840:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1113.141449] LustreError: 92840:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1113.289077] Lustre: server umount lustre-OST0000 complete [ 1117.303856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.480766] LustreError: 93469:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1117.536618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1118.906132] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1121.906314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1122.126227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1123.452376] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1125.317091] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1126.600743] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1128.801337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1131.133780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1131.156240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1131.221932] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1131.224953] LustreError: 93457:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1131.234782] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1131.237330] LustreError: 95331:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -98 [ 1131.239292] LustreError: 95331:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 1131.241024] LustreError: 95331:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1131.289345] Lustre: server umount lustre-OST0000 complete [ 1131.290885] LustreError: 95331:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1134.118359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1134.189993] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1134.193067] LustreError: 93458:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1134.203746] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1134.206896] LustreError: 95796:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -98 [ 1134.209237] LustreError: 95796:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 1134.211471] LustreError: 95796:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1134.263507] Lustre: server umount lustre-OST0000 complete [ 1134.265464] LustreError: 95796:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1135.308512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1137.575985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1137.659221] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1137.661582] Lustre: Found index 0 for lustre-OST0000, updating log [ 1137.664404] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1137.691108] Lustre: lustre-OST0000: new disk, initializing [ 1137.693511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1137.808327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1140.159848] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1142.408246] Lustre: Failing over lustre-OST0000 [ 1142.442152] LustreError: 96875:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1142.444322] LustreError: 96875:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1142.547447] Lustre: server umount lustre-OST0000 complete [ 1146.529768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.530481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1146.534775] Lustre: Skipped 1 previous similar message [ 1146.537363] Lustre: Skipped 2 previous similar messages [ 1150.599645] LustreError: 97076:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1150.601835] LustreError: 97076:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1150.643454] Lustre: server umount lustre-MDT0000 complete [ 1151.969939] LustreError: 93468:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1151.977101] LustreError: 93468:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 1152.561841] LustreError: 95351:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761284317 with bad export cookie 4236484533288374368 [ 1152.569946] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1152.780251] Lustre: server umount lustre-MDT0001 complete [ 1156.507323] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:38:40 (1761284320) [ 1161.702793] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1165.174326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1165.298574] LustreError: 98643:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1165.324935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1166.562642] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1169.347947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1170.824071] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1171.787307] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1174.007548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1176.041984] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1178.871799] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1178.902993] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1178.927509] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1178.963467] Lustre: lustre-OST0001: new disk, initializing [ 1178.965770] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1180.975353] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1181.066298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1181.069478] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1181.107970] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1184.084094] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1188.761456] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1196.513747] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.515880] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1196.517393] Lustre: Skipped 1 previous similar message [ 1196.520289] Lustre: Skipped 2 previous similar messages [ 1201.275271] LustreError: 101886:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 1201.278258] LustreError: 101886:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1201.301083] Lustre: server umount lustre-MDT0001 complete [ 1201.634322] LustreError: 101383:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1201.641173] LustreError: 101383:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 1202.352356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1204.687220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1204.719060] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1204.720948] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1204.737881] Lustre: 98633:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1204.988284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1204.990934] Lustre: Skipped 3 previous similar messages [ 1205.990126] LustreError: 102593:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1205.992670] LustreError: 102593:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1205.995609] LustreError: 98635:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1206.024313] LustreError: 102602:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.138@tcp: cfg command failed: rc = -17 [ 1206.026967] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1206.026967] [ 1206.292369] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1209.364294] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 01:39:33 (1761284373) [ 1210.342696] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1214.926707] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1220.919778] Lustre: 104490:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1228.911879] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1230.121542] Lustre: 105622:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1235.210763] Lustre: 100128:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 506 < left 1007, rollback = 2 [ 1235.215804] Lustre: 100128:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 4/16/6, destroy: 0/0/0 [ 1235.219217] Lustre: 100128:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 8/1007/0 [ 1235.222974] Lustre: 100128:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [ 1235.226102] Lustre: 100128:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 12/236/4, delete: 0/0/0 [ 1235.229500] Lustre: 100128:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0 [ 1246.944847] Lustre: 98637:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 508 < left 993, rollback = 2 [ 1246.948849] Lustre: 98637:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 8 previous similar messages [ 1246.951143] Lustre: 98637:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/5, destroy: 0/0/0 [ 1246.953168] Lustre: 98637:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 1246.955307] Lustre: 98637:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 7/993/0 [ 1246.957537] Lustre: 98637:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 1246.959723] Lustre: 98637:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 5/58/0, punch: 0/0/0, quota 1/3/0 [ 1246.961930] Lustre: 98637:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 1246.964150] Lustre: 98637:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 10/203/3, delete: 0/0/0 [ 1246.966476] Lustre: 98637:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 1246.969229] Lustre: 98637:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0 [ 1246.972214] Lustre: 98637:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 1281.503699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1281.506511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1281.510514] Lustre: Skipped 1 previous similar message [ 1281.512264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1281.514330] Lustre: Skipped 1 previous similar message [ 1287.136281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1287.138052] Lustre: Skipped 7 previous similar messages [ 1287.503857] LustreError: 106056:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1287.505902] LustreError: 106056:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1287.529180] Lustre: server umount lustre-MDT0000 complete [ 1288.742735] LustreError: 100912:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761284453 with bad export cookie 4236484533288375278 [ 1288.743995] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1288.746132] LustreError: 100912:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1288.900086] Lustre: server umount lustre-MDT0001 complete [ 1299.472772] Lustre: server umount lustre-OST0000 complete [ 1311.160728] Lustre: server umount lustre-OST0001 complete [ 1312.965366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1316.059427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1316.409449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1319.447750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1322.412349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1322.787382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1329.477909] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1333.130607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1333.136674] Lustre: lustre-MDT0000: reset Object Index mappings [ 1333.293653] LustreError: 109536:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1333.300339] LustreError: 109536:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 1333.321264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1334.651286] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1337.455886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1337.462677] Lustre: lustre-MDT0001: reset Object Index mappings [ 1339.113851] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1340.117352] Lustre: 110652:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1342.371788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1343.657258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 1344.534068] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1347.574212] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1347.755409] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 1349.300097] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 1349.304690] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 1349.969498] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1353.933939] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1356.709847] Lustre: 112505:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1401.657374] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 01:42:44 (1761284564) [ 1429.988327] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1429.988404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1429.994272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1430.000553] Lustre: Skipped 5 previous similar messages [ 1432.368384] LustreError: 112854:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1432.371254] LustreError: 112854:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 1432.437466] Lustre: server umount lustre-MDT0000 complete [ 1434.801977] LustreError: 110655:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761284599 with bad export cookie 4236484533288421996 [ 1434.805399] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1434.811391] LustreError: 110655:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1435.104617] LustreError: 109532:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1435.114718] LustreError: 109532:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 1441.254226] Lustre: server umount lustre-MDT0001 complete [ 1449.785706] Lustre: server umount lustre-OST0000 complete [ 1466.335382] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1466.492832] Lustre: server umount lustre-OST0001 complete [ 1470.190612] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 1475.834825] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1481.509571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1485.305906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1489.154795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1493.422519] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1498.012088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1498.074740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1498.245800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1498.273449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1498.326421] Lustre: lustre-MDT0000: new disk, initializing [ 1498.388430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1498.393328] Lustre: Skipped 3 previous similar messages [ 1498.401958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1500.933651] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1505.299925] LustreError: 116213:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1505.303820] LustreError: 116213:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 1505.447289] Lustre: server umount lustre-MDT0000 complete [ 1507.038218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1511.184497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1511.228980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1511.410540] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1511.419772] Lustre: 116733:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1511.428081] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1514.024891] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1516.978145] Lustre: server umount lustre-MDT0000 complete [ 1521.958484] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 01:44:45 (1761284685) [ 1522.760240] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1523.792124] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 01:44:47 (1761284687) [ 1533.428383] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 1538.905970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1541.021709] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1546.323523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1546.378648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1546.475743] Lustre: 118753:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1546.526583] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1546.529478] Lustre: Skipped 1 previous similar message [ 1546.581096] Lustre: lustre-MDT0001: new disk, initializing [ 1546.700013] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1546.716846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1546.770466] Lustre: 119492:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 1546.780063] Lustre: 119492:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 8 previous similar messages [ 1546.784760] Lustre: 119492:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 1546.789656] Lustre: 119492:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 1546.797088] Lustre: 119492:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 1546.805821] Lustre: 119492:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 1546.814592] Lustre: 119492:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 1546.821246] Lustre: 119492:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 1546.828252] Lustre: 119492:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 1546.831588] Lustre: 119492:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 1546.835144] Lustre: 119492:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 1546.838549] Lustre: 119492:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 1549.931701] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1553.992615] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1558.582493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1558.637247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1558.801644] Lustre: lustre-OST0000: new disk, initializing [ 1558.805443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1560.058359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1560.066232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1560.138326] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1562.724102] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1570.668522] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1570.719793] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1570.867735] Lustre: lustre-OST0001: new disk, initializing [ 1570.887740] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1572.910184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1572.919826] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1573.077158] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1574.765338] hrtimer: interrupt took 6469225 ns [ 1578.516357] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 1593.597778] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1600.215726] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1605.792798] Lustre: 118759:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 504 < left 1021, rollback = 2 [ 1605.819853] Lustre: 118759:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 1605.848183] Lustre: 118759:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 4/16/8, destroy: 0/0/0 [ 1605.869728] Lustre: 118759:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1605.884784] Lustre: 118759:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 9/1021/0 [ 1605.904554] Lustre: 118759:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1605.914167] Lustre: 118759:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 7/153/0, punch: 0/0/0, quota 1/3/0 [ 1605.938308] Lustre: 118759:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1605.944447] Lustre: 118759:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 13/232/4, delete: 0/0/0 [ 1605.955742] Lustre: 118759:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1605.965768] Lustre: 118759:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 5/5/0, ref_del: 0/0/0 [ 1605.978852] Lustre: 118759:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1993.762481] Lustre: 127557:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1993.766674] Lustre: 127557:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 8 previous similar messages [ 1993.769645] Lustre: 127557:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1993.772533] Lustre: 127557:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 1993.775583] Lustre: 127557:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1993.778610] Lustre: 127557:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 1993.781435] Lustre: 127557:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1993.828900] Lustre: 127557:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 1993.842554] Lustre: 127557:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1993.865396] Lustre: 127557:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 1993.888190] Lustre: 127557:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1993.909650] Lustre: 127557:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2271.778460] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:57:14 (1761285434) [ 2289.632339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2289.634898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.652558] Lustre: Skipped 3 previous similar messages [ 2289.663231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2289.666292] Lustre: Skipped 7 previous similar messages [ 2292.381246] LustreError: 127794:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2292.409614] LustreError: 127794:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2292.628168] Lustre: server umount lustre-MDT0000 complete [ 2295.265900] LustreError: 121961:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2295.321712] LustreError: 121961:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [ 2297.182731] LustreError: 119931:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761285461 with bad export cookie 4236484533288485577 [ 2297.186146] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2297.202305] LustreError: 119931:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2300.383973] LustreError: 121947:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2300.388491] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2300.394618] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2300.394628] Lustre: Skipped 4 previous similar messages [ 2300.406805] LustreError: 121947:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2300.411320] Lustre: Skipped 4 previous similar messages [ 2303.875724] Lustre: server umount lustre-MDT0001 complete [ 2314.063129] LustreError: 128197:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2314.071040] LustreError: 128197:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 2314.237646] Lustre: server umount lustre-OST0000 complete [ 2324.549728] Lustre: server umount lustre-OST0001 complete [ 2330.751333] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 2341.228631] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 2351.462651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2358.569645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2364.238195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2370.416737] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2377.458211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2377.548360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2377.860993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2377.912260] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2378.010858] Lustre: lustre-MDT0000: new disk, initializing [ 2378.097172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2378.100259] Lustre: Skipped 5 previous similar messages [ 2378.117862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2382.220956] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2393.208671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2393.297093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2393.426917] Lustre: 130587:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2393.525849] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2393.529176] Lustre: Skipped 1 previous similar message [ 2393.608728] Lustre: lustre-MDT0001: new disk, initializing [ 2393.764950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2393.774225] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2393.906332] Lustre: 131384:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 2393.922322] Lustre: 131384:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 255 previous similar messages [ 2393.939146] Lustre: 131384:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 2393.954549] Lustre: 131384:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 255 previous similar messages [ 2393.967263] Lustre: 131384:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 2393.974028] Lustre: 131384:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 255 previous similar messages [ 2393.985925] Lustre: 131384:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 2393.997154] Lustre: 131384:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 255 previous similar messages [ 2394.017427] Lustre: 131384:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 2394.031359] Lustre: 131384:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 255 previous similar messages [ 2394.049407] Lustre: 131384:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 2394.061054] Lustre: 131384:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 255 previous similar messages [ 2397.462487] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2404.372818] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2407.812299] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2414.712396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2414.791544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2415.087299] Lustre: lustre-OST0000: new disk, initializing [ 2415.090968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2415.187865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2415.199351] Lustre: Skipped 1 previous similar message [ 2416.625779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2416.632704] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2416.796079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2421.238175] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 2428.657482] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2438.210573] Lustre: lustre-MDD0000: changelog on [ 2440.982768] Lustre: lustre-MDD0001: changelog on [ 2444.799310] Lustre: 130593:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 508 < left 539, rollback = 2 [ 2444.806313] Lustre: 130593:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 2444.812788] Lustre: 130593:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 2444.820618] Lustre: 130593:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2444.824316] Lustre: 130593:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 9/539/0 [ 2444.836406] Lustre: 130593:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2444.844777] Lustre: 130593:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 10/93/1, punch: 0/0/0, quota 1/3/0 [ 2444.860205] Lustre: 130593:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2444.867129] Lustre: 130593:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 4/67/1, delete: 0/0/0 [ 2444.878580] Lustre: 130593:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2444.888196] Lustre: 130593:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2444.894943] Lustre: 130593:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2476.800284] Lustre: 130594:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 525, rollback = 2 [ 2476.805237] Lustre: 130594:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 7532 previous similar messages [ 2476.826394] Lustre: 130594:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 2/8/0, destroy: 0/0/0 [ 2476.830060] Lustre: 130594:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 7535 previous similar messages [ 2476.833827] Lustre: 130594:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 2476.838269] Lustre: 130594:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 7535 previous similar messages [ 2476.856488] Lustre: 130594:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 7/39/0, punch: 0/0/0, quota 1/3/0 [ 2476.860605] Lustre: 130594:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 7538 previous similar messages [ 2476.879462] Lustre: 130594:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 3/50/0, delete: 0/0/0 [ 2476.886152] Lustre: 130594:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 7541 previous similar messages [ 2476.890441] Lustre: 130594:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2476.894065] Lustre: 130594:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 7541 previous similar messages [ 2549.901940] Lustre: 130593:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 508 < left 525, rollback = 2 [ 2549.906318] Lustre: 130593:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 5966 previous similar messages [ 2549.913623] Lustre: 130593:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 2/8/6, destroy: 0/0/0 [ 2549.921887] Lustre: 130593:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 5963 previous similar messages [ 2549.929424] Lustre: 130593:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 2549.953087] Lustre: 130593:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 5963 previous similar messages [ 2549.965556] Lustre: 130593:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 7/39/1, punch: 0/0/0, quota 1/3/0 [ 2549.978657] Lustre: 130593:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 5960 previous similar messages [ 2549.985076] Lustre: 130593:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 3/50/1, delete: 0/0/0 [ 2549.996328] Lustre: 130593:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 5957 previous similar messages [ 2550.007785] Lustre: 130593:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2550.017170] Lustre: 130593:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 5957 previous similar messages [ 2677.915113] Lustre: 130593:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 525, rollback = 2 [ 2677.930046] Lustre: 130593:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 19139 previous similar messages [ 2677.944246] Lustre: 130593:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 2/8/0, destroy: 0/0/0 [ 2677.955290] Lustre: 130593:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 19139 previous similar messages [ 2677.972832] Lustre: 130593:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 2677.977074] Lustre: 130593:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 19139 previous similar messages [ 2677.981677] Lustre: 130593:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 7/39/0, punch: 0/0/0, quota 1/3/0 [ 2677.987824] Lustre: 130593:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 19139 previous similar messages [ 2677.993374] Lustre: 130593:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 3/50/0, delete: 0/0/0 [ 2677.998356] Lustre: 130593:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 19139 previous similar messages [ 2678.019180] Lustre: 130593:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2678.024370] Lustre: 130593:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 19142 previous similar messages [ 2958.035985] Lustre: 130595:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 525, rollback = 2 [ 2958.043813] Lustre: 130595:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 34859 previous similar messages [ 2958.051379] Lustre: 130595:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 2/8/0, destroy: 0/0/0 [ 2958.062162] Lustre: 130595:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 34859 previous similar messages [ 2958.077825] Lustre: 130595:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 2958.081304] Lustre: 130595:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 34859 previous similar messages [ 2958.086951] Lustre: 130595:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 7/39/1, punch: 0/0/0, quota 1/3/0 [ 2958.100038] Lustre: 130595:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 34859 previous similar messages [ 2958.104138] Lustre: 130595:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 3/50/1, delete: 0/0/0 [ 2958.108858] Lustre: 130595:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 34859 previous similar messages [ 2958.120385] Lustre: 130595:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2958.125154] Lustre: 130595:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 34856 previous similar messages [ 3504.764799] Lustre: 133696:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 525, rollback = 2 [ 3504.776342] Lustre: 133696:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 80999 previous similar messages [ 3504.793743] Lustre: 133696:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 2/8/0, destroy: 0/0/0 [ 3504.799294] Lustre: 133696:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 80999 previous similar messages [ 3504.804338] Lustre: 133696:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 3504.814787] Lustre: 133696:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 80999 previous similar messages [ 3504.820790] Lustre: 133696:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 7/39/1, punch: 0/0/0, quota 1/3/0 [ 3504.827736] Lustre: 133696:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 80999 previous similar messages [ 3504.831965] Lustre: 133696:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 3/50/1, delete: 0/0/0 [ 3504.838320] Lustre: 133696:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 80999 previous similar messages [ 3504.843268] Lustre: 133696:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3504.847983] Lustre: 133696:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 80999 previous similar messages [ 4104.775555] Lustre: 130594:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 525, rollback = 2 [ 4104.797346] Lustre: 130594:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 93809 previous similar messages [ 4104.819587] Lustre: 130594:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 2/8/0, destroy: 0/0/0 [ 4104.832716] Lustre: 130594:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 93809 previous similar messages [ 4104.838077] Lustre: 130594:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 4104.860248] Lustre: 130594:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 93809 previous similar messages [ 4104.868823] Lustre: 130594:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 7/39/0, punch: 0/0/0, quota 1/3/0 [ 4104.872149] Lustre: 130594:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 93809 previous similar messages [ 4104.882308] Lustre: 130594:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 3/50/0, delete: 0/0/0 [ 4104.885838] Lustre: 130594:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 93809 previous similar messages [ 4104.890013] Lustre: 130594:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4104.897134] Lustre: 130594:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 93809 previous similar messages [ 4162.090226] Lustre: lustre-MDD0001: changelog off [ 4163.849391] Lustre: 130595:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4164.564930] Lustre: lustre-MDD0000: changelog off [ 4172.774608] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4172.780246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4172.799675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4172.799684] Lustre: Skipped 3 previous similar messages [ 4177.890757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4177.893380] Lustre: Skipped 2 previous similar messages [ 4183.007282] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4183.010780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4183.019534] Lustre: Skipped 1 previous similar message [ 4183.053476] LustreError: 140392:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4183.066099] LustreError: 140392:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4183.243898] Lustre: server umount lustre-OST0000 complete [ 4188.129592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4188.131450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4188.150466] Lustre: Skipped 2 previous similar messages [ 4188.158465] Lustre: Skipped 1 previous similar message [ 4192.243511] LustreError: 140594:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4192.251657] LustreError: 140594:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4192.400810] Lustre: server umount lustre-MDT0000 complete [ 4193.248459] LustreError: 131460:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4193.262834] LustreError: 131460:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 4195.711658] LustreError: 130580:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761287360 with bad export cookie 4236484533289293314 [ 4195.715842] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4195.736990] LustreError: 130580:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4196.216263] Lustre: server umount lustre-MDT0001 complete [ 4205.760910] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 4208.314955] Key type lgssc unregistered [ 4208.630386] LNet: 141326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4208.639119] LNetError: 141326:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4208.659719] LNet: Removed LNI 192.168.202.138@tcp [ 4209.399635] Key type .llcrypt unregistered [ 4209.401214] Key type ._llcrypt unregistered [ 4225.317772] Key type ._llcrypt registered [ 4225.319605] Key type .llcrypt registered [ 4225.418037] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 4239.436917] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 4240.435809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4240.484753] alg: No test for adler32 (adler32-zlib) [ 4241.507803] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 4241.735418] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 4243.439806] Key type lgssc registered [ 4244.444813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4252.434564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4257.968530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4263.448563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4268.798570] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4274.889591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4274.945768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4274.957704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4276.224050] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4276.273990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4276.416887] Lustre: lustre-MDT0000: new disk, initializing [ 4276.542558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4276.560597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4280.773857] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4291.781148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4291.891858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4292.031902] Lustre: 144506:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4292.094222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4292.102183] Lustre: Skipped 1 previous similar message [ 4292.177835] Lustre: lustre-MDT0001: new disk, initializing [ 4292.289240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4292.320369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4292.334871] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4292.478745] Lustre: 145303:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 4292.519854] Lustre: 145303:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 4292.548490] Lustre: 145303:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 4292.563785] Lustre: 145303:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 4292.579224] Lustre: 145303:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 4292.591857] Lustre: 145303:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 4296.665417] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4304.169404] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4307.774854] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4314.060199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4314.106803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4314.301250] Lustre: lustre-OST0000: new disk, initializing [ 4314.303924] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4314.400837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4319.480440] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4322.362172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4322.366739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4322.453995] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4325.588618] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4330.671975] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_module obdecho/obdecho [ 4332.304520] LustreError: 147130:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4332.308371] LustreError: 147130:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 4333.780701] LustreError: 147227:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 4343.267126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4343.272191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4343.281031] Lustre: Skipped 2 previous similar messages [ 4343.288030] Lustre: Skipped 2 previous similar messages [ 4347.970156] LustreError: 147567:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4348.062986] Lustre: server umount lustre-MDT0000 complete [ 4350.863020] LustreError: 144498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761287515 with bad export cookie 5876271098094059033 [ 4350.872194] LustreError: 144498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4350.872577] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4350.886971] LustreError: 147768:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007146be0b x1846843703074048/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4350.906262] LustreError: 147768:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x20000000a:0x0:0x0]: rc = -5 [ 4350.913946] LustreError: 147768:0:(llog_cat.c:443:llog_cat_close()) lustre-MDT0000-osp-MDT0001: failure destroying log during cleanup: rc = -5 [ 4351.029289] LustreError: 147768:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4351.033110] LustreError: 147768:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4351.199931] Lustre: server umount lustre-MDT0001 complete [ 4364.418555] LustreError: 147968:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4364.424641] LustreError: 147968:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4364.644684] Lustre: server umount lustre-OST0000 complete [ 4384.720468] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 4393.640705] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 4401.356662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4407.188200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4412.294296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4417.594433] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4423.721301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4423.783184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4423.970245] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4424.007758] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4424.116272] Lustre: lustre-MDT0000: new disk, initializing [ 4424.229182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4424.246525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4427.734791] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4435.755696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4435.811759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4435.863878] Lustre: 150916:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4435.917958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4435.920347] Lustre: Skipped 1 previous similar message [ 4435.969179] Lustre: lustre-MDT0001: new disk, initializing [ 4436.064539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4436.130414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4436.145334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4436.217661] Lustre: 151712:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 4436.228721] Lustre: 151712:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 4436.246253] Lustre: 151712:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 4436.253115] Lustre: 151712:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4436.261174] Lustre: 151712:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 4436.278825] Lustre: 151712:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4436.295014] Lustre: 151712:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 4436.312245] Lustre: 151712:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4436.320868] Lustre: 151712:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 4436.328912] Lustre: 151712:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4436.340306] Lustre: 151712:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 4436.351257] Lustre: 151712:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4439.640784] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4445.284329] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4448.133190] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4454.677584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4454.739581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4454.959356] Lustre: lustre-OST0000: new disk, initializing [ 4454.972524] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4455.038763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4460.726898] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4462.161585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4462.171939] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4462.304256] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4467.738190] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4471.899767] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4472.096199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4475.590779] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4475.805323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4477.925941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4477.926934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4477.928490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4477.938034] Lustre: Skipped 1 previous similar message [ 4477.941526] LustreError: Skipped 1 previous similar message [ 4483.046610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4483.049572] Lustre: Skipped 1 previous similar message [ 4483.522130] LustreError: 153855:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4483.529571] LustreError: 153855:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4483.645222] Lustre: server umount lustre-OST0000 complete [ 4487.141195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4487.145064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4487.154087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4487.156578] Lustre: Skipped 1 previous similar message [ 4492.854533] LustreError: 154056:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4492.869902] LustreError: 154056:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4493.046451] Lustre: server umount lustre-MDT0000 complete [ 4493.284677] LustreError: 150927:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4493.297651] LustreError: 150927:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [ 4496.358329] LustreError: 150909:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761287660 with bad export cookie 5876271098094060034 [ 4496.365336] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4496.373439] LustreError: 150909:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4496.724398] Lustre: server umount lustre-MDT0001 complete [ 4500.176228] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:34:23 (1761287663) [ 4501.573500] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4503.209624] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 02:34:26 (1761287666) [ 4509.738940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4510.084573] LustreError: 154930:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4510.166469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4513.524318] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4515.297065] LustreError: 154930:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4519.396564] LustreError: 154930:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4520.816552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4524.792355] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4530.049507] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4534.060368] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4540.910780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4547.514481] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4553.073636] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4559.337484] Lustre: 154923:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 504 < left 1021, rollback = 2 [ 4559.346732] Lustre: 154923:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 4559.354416] Lustre: 154923:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 4/16/8, destroy: 0/0/0 [ 4559.361525] Lustre: 154923:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4559.368464] Lustre: 154923:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 9/1021/0 [ 4559.376797] Lustre: 154923:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4559.380423] Lustre: 154923:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 7/153/0, punch: 0/0/0, quota 1/3/0 [ 4559.397711] Lustre: 154923:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4559.404247] Lustre: 154923:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 13/232/4, delete: 0/0/0 [ 4559.423708] Lustre: 154923:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4559.427248] Lustre: 154923:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 5/5/0, ref_del: 0/0/0 [ 4559.452928] Lustre: 154923:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4564.966663] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4564.980219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4564.982986] Lustre: Skipped 2 previous similar messages [ 4564.988156] Lustre: Skipped 4 previous similar messages [ 4570.291398] LustreError: 157180:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4570.303478] LustreError: 157180:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4570.459242] Lustre: server umount lustre-MDT0000 complete [ 4574.086080] LustreError: 154923:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4574.405318] LustreError: 154910:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761287739 with bad export cookie 5876271098094060958 [ 4574.417220] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4574.437918] LustreError: 154910:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4575.201108] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4575.222338] Lustre: Skipped 1 previous similar message [ 4575.236607] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4575.243696] Lustre: Skipped 4 previous similar messages [ 4580.891434] Lustre: server umount lustre-MDT0001 complete [ 4587.444917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4599.559784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4599.740071] LustreError: 158465:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4599.772222] LustreError: 158465:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 4599.792729] LustreError: 158457:0:(llog.c:1616:llog_backup()) MGC192.168.202.138@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4599.797344] Lustre: 158457:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.202.138@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4620.153249] LustreError: 158480:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4620.183871] LustreError: 158480:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 12 previous similar messages [ 4654.986088] LustreError: 158464:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4654.998089] LustreError: 158464:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 26 previous similar messages [ 4661.216486] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4661.223732] Lustre: 158457:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.202.138@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4690.911430] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4690.919982] Lustre: 154921:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.202.138@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4702.201214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4702.224993] Lustre: Skipped 2 previous similar messages [ 4707.307059] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4707.739276] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4711.391269] LustreError: 154921:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 4716.999382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4719.480391] LustreError: 158464:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4719.498529] LustreError: 158464:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 44 previous similar messages [ 4720.609774] LustreError: 142713:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002f38ad80 x1846843703165184/t0(0) o250->MGC192.168.202.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4724.691293] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4726.249394] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4729.950144] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4731.566754] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4734.826571] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4736.499928] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4756.965420] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4756.966919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4756.975429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4756.975438] Lustre: Skipped 2 previous similar messages [ 4756.975952] Lustre: Skipped 1 previous similar message [ 4759.259649] LustreError: 159964:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4759.265503] LustreError: 159964:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 4759.337214] Lustre: server umount lustre-MDT0000 complete [ 4763.019438] LustreError: 154909:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761287927 with bad export cookie 5876271098094067433 [ 4763.028233] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4763.037666] LustreError: 154909:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4763.573588] Lustre: server umount lustre-MDT0001 complete [ 4776.828732] Lustre: server umount lustre-OST0000 complete [ 4795.474724] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 4803.868605] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 4812.405739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4818.337873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4823.472617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4829.327161] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4835.313195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4835.380605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4835.663082] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4835.715788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4835.808943] Lustre: lustre-MDT0000: new disk, initializing [ 4835.914769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4835.925974] Lustre: Skipped 1 previous similar message [ 4835.947893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4839.693548] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4849.825502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4849.894225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4850.001673] Lustre: 163315:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4850.097221] LustreError: 163327:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4850.120224] LustreError: 163327:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 6 previous similar messages [ 4850.385047] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4850.476851] Lustre: 164116:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 4850.487743] Lustre: 164116:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 3 previous similar messages [ 4850.503084] Lustre: 164116:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 4850.524976] Lustre: 164116:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 4850.528052] Lustre: 164116:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 4850.535563] Lustre: 164116:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 4850.551554] Lustre: 164116:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 4850.576648] Lustre: 164116:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 4850.587250] Lustre: 164116:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 4850.602875] Lustre: 164116:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 4850.617288] Lustre: 164116:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 4850.620871] Lustre: 164116:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 4854.017454] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4862.244873] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4865.365441] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4871.824656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4871.877811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4872.121613] Lustre: lustre-OST0000: new disk, initializing [ 4872.124154] Lustre: Skipped 1 previous similar message [ 4872.135504] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4872.138232] Lustre: Skipped 2 previous similar messages [ 4878.420833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4878.427967] Lustre: Skipped 1 previous similar message [ 4878.441391] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4878.593345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4878.837703] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4885.934483] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4890.149520] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4890.405234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4894.325381] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4894.487593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4899.304076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4899.311199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4899.326329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4899.334232] Lustre: Skipped 3 previous similar messages [ 4902.080394] LustreError: 166255:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4902.083318] LustreError: 166255:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4902.291190] Lustre: server umount lustre-OST0000 complete [ 4906.985350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4912.130259] Lustre: server umount lustre-MDT0000 complete [ 4916.157311] LustreError: 163307:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288080 with bad export cookie 5876271098094084254 [ 4916.162131] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4916.194865] LustreError: 163307:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4916.604044] Lustre: server umount lustre-MDT0001 complete [ 4924.235250] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 02:41:27 (1761288087) [ 4930.131953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4933.714334] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4941.212818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4944.612296] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4949.610779] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4952.777932] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4958.758694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4964.114897] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 4968.746702] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4988.582436] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 02:42:32 (1761288152) [ 4992.998876] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4993.001825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4993.005801] Lustre: Skipped 3 previous similar messages [ 4993.013127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4993.018206] Lustre: Skipped 3 previous similar messages [ 4997.687241] LustreError: 169794:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4997.696479] LustreError: 169794:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4997.833468] Lustre: server umount lustre-OST0000 complete [ 5007.016929] Lustre: server umount lustre-MDT0000 complete [ 5010.060795] LustreError: 167310:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288174 with bad export cookie 5876271098094085178 [ 5010.065521] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5010.091955] LustreError: 167310:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5010.390797] Lustre: server umount lustre-MDT0001 complete [ 5018.912217] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 5021.374519] Key type lgssc unregistered [ 5021.668712] LNet: 170740:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5021.672800] LNetError: 170740:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5022.694357] LNet: Removed LNI 192.168.202.138@tcp [ 5023.411252] Key type .llcrypt unregistered [ 5023.413423] Key type ._llcrypt unregistered [ 5038.044723] Key type ._llcrypt registered [ 5038.046209] Key type .llcrypt registered [ 5038.143348] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5038.780415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5038.823034] alg: No test for adler32 (adler32-zlib) [ 5039.929397] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 5040.131597] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 5041.783196] Key type lgssc registered [ 5042.490707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5047.215677] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5047.239848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5062.751341] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5067.871284] LustreError: 171941:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.138@tcp: failed processing log, type 4: rc = -110 [ 5072.991533] LustreError: 171941:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5072.998778] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5073.003567] LustreError: MGC192.168.202.138@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5073.012445] LustreError: 171941:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5073.018878] LustreError: 171941:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -22 [ 5073.025611] LustreError: 171941:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 5073.034210] LustreError: 171941:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 5073.100153] Lustre: server umount lustre-OST0000 complete [ 5073.102093] LustreError: 171941:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5078.952360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5080.340900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5083.298238] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5089.310523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5089.546288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5092.019400] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5095.132949] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5097.232459] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5101.053598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5101.158156] LustreError: 174036:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5101.163428] Lustre: 174036:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.202.138@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5101.266624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5104.667711] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5107.966965] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5110.241209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5110.244928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5110.252703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5110.755887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5110.763749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5110.770683] Lustre: Skipped 1 previous similar message [ 5114.849693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5114.857582] Lustre: Skipped 2 previous similar messages [ 5115.345374] LustreError: 174687:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5115.349577] LustreError: 174687:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5115.405337] Lustre: server umount lustre-MDT0000 complete [ 5117.444858] LustreError: 174688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288282 with bad export cookie 14369734935727608009 [ 5117.447424] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5117.449899] LustreError: 174688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5118.943255] LustreError: 173157:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000242ae818 x1846844539820800/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5118.960914] LustreError: 174889:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5118.964140] LustreError: 174889:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5119.065526] Lustre: server umount lustre-MDT0001 complete [ 5131.383448] LustreError: 175091:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5131.386661] LustreError: 175091:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5131.484748] Lustre: server umount lustre-OST0000 complete [ 5136.175340] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 5140.813689] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5145.464901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5148.612134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5151.658375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5154.757510] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5157.995165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5158.028399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5158.136315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5158.156235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5158.198658] Lustre: lustre-MDT0000: new disk, initializing [ 5158.243201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5158.257099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5160.106451] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5165.210652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5165.242113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5165.289923] Lustre: 177428:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5165.320909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5165.323803] Lustre: Skipped 1 previous similar message [ 5165.367422] Lustre: lustre-MDT0001: new disk, initializing [ 5165.419192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5165.438210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5165.443166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5165.497086] Lustre: 178227:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 5165.501378] Lustre: 178227:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 5165.506162] Lustre: 178227:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 5165.510953] Lustre: 178227:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 5165.515446] Lustre: 178227:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 5165.519178] Lustre: 178227:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 5167.165547] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5171.264462] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5172.951146] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5176.170373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5176.220578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5176.363233] Lustre: lustre-OST0000: new disk, initializing [ 5176.366055] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5176.417242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5179.094978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5179.098915] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5179.163608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5179.197206] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5183.036928] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5184.917042] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5185.013862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5186.942981] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5187.048301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5189.602300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5189.604038] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5189.605744] LustreError: Skipped 1 previous similar message [ 5189.611180] Lustre: Skipped 1 previous similar message [ 5189.614383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5189.618499] Lustre: Skipped 1 previous similar message [ 5194.721106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5194.724289] Lustre: Skipped 1 previous similar message [ 5202.400622] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5202.441903] LustreError: 180371:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5202.445430] LustreError: 180371:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5202.535159] Lustre: server umount lustre-OST0000 complete [ 5204.960633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5204.961153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5204.966415] Lustre: Skipped 1 previous similar message [ 5204.971013] Lustre: Skipped 3 previous similar messages [ 5210.636808] LustreError: 180574:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5210.639617] LustreError: 180574:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5210.701850] Lustre: server umount lustre-MDT0000 complete [ 5212.682313] LustreError: 177421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288377 with bad export cookie 14369734935727608730 [ 5212.686737] LustreError: 177421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5212.687553] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5212.869993] Lustre: server umount lustre-MDT0001 complete [ 5217.314676] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 02:46:21 (1761288381) [ 5230.216616] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 5233.781311] Key type lgssc unregistered [ 5234.110667] LNet: 181958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5234.125473] LNetError: 181958:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5234.141438] LNet: Removed LNI 192.168.202.138@tcp [ 5235.274856] Key type .llcrypt unregistered [ 5235.276233] Key type ._llcrypt unregistered [ 5257.904200] Key type ._llcrypt registered [ 5257.910507] Key type .llcrypt registered [ 5258.038032] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5259.026579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5259.039823] alg: No test for adler32 (adler32-zlib) [ 5260.372933] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 5260.611548] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 5262.352501] Key type lgssc registered [ 5263.902574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5272.783954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5272.795979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5274.129960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5277.084536] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5284.324678] LustreError: 183157:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5284.344207] LustreError: 183157:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 5284.496363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5284.764354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5287.974656] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5292.096112] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5294.751375] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5300.399665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5300.568271] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5300.570131] LustreError: 184798:0:(llog.c:1657:llog_backup()) MGC192.168.202.138@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5300.574788] Lustre: 184798:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.202.138@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5300.667641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5304.917885] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5309.563133] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5311.319834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5311.325913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5311.333647] Lustre: Skipped 2 previous similar messages [ 5311.341616] Lustre: Skipped 1 previous similar message [ 5316.579923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5316.582944] Lustre: Skipped 3 previous similar messages [ 5317.553129] LustreError: 185452:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5317.652419] Lustre: server umount lustre-MDT0000 complete [ 5320.784690] LustreError: 183137:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288485 with bad export cookie 15238989592436414903 [ 5320.790569] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5320.803352] LustreError: 183137:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5320.979047] LustreError: 185653:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5320.992472] LustreError: 185653:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5321.245907] Lustre: server umount lustre-MDT0001 complete [ 5334.971744] LustreError: 185855:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5334.978417] LustreError: 185855:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5335.118455] Lustre: server umount lustre-OST0000 complete [ 5342.909877] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 5350.841543] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5359.148816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5364.908058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5370.324288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5376.356265] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5383.011894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5383.089472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5383.374244] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5383.420784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5383.545192] Lustre: lustre-MDT0000: new disk, initializing [ 5383.670257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5383.697494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5387.495576] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5396.544418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5396.605697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5396.667112] Lustre: 188194:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5396.723256] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5396.725043] Lustre: Skipped 1 previous similar message [ 5396.730000] LustreError: 188963:0:(llog.c:1657:llog_backup()) MGC192.168.202.138@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5396.734712] LustreError: 188963:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 5396.739782] Lustre: 188963:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.202.138@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5396.754340] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5396.757222] Lustre: Skipped 1 previous similar message [ 5396.825554] Lustre: lustre-MDT0001: new disk, initializing [ 5396.927778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5396.972245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5396.990695] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5397.067224] Lustre: 188992:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 5397.075141] Lustre: 188992:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 5397.081910] Lustre: 188992:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 5397.091632] Lustre: 188992:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 5397.095833] Lustre: 188992:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 5397.103918] Lustre: 188992:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 5400.538411] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5406.908119] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5409.786194] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5415.050181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5415.129425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5415.309526] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5415.311382] Lustre: Skipped 1 previous similar message [ 5415.314302] LustreError: 189917:0:(llog.c:1657:llog_backup()) MGC192.168.202.138@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5415.317645] LustreError: 189917:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 5415.322584] Lustre: 189917:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.202.138@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5415.329082] Lustre: 189917:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5415.358662] Lustre: lustre-OST0000: new disk, initializing [ 5415.369792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5415.455765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5419.846924] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5423.169454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5423.178055] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5423.273983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5425.828177] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5429.288288] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5429.471121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5432.387157] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5432.576238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5438.949464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5438.951490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5438.952780] LustreError: Skipped 1 previous similar message [ 5438.962197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5438.969303] Lustre: Skipped 1 previous similar message [ 5440.125275] LustreError: 191134:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5440.140787] LustreError: 191134:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5440.255551] Lustre: server umount lustre-OST0000 complete [ 5443.040164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5443.043865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5443.050657] Lustre: Skipped 1 previous similar message [ 5443.054157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5448.732545] LustreError: 191335:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5448.735346] LustreError: 191335:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5448.819447] Lustre: server umount lustre-MDT0000 complete [ 5449.190379] LustreError: 188205:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5449.210547] LustreError: 188205:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 5451.811676] LustreError: 188976:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288616 with bad export cookie 15238989592436415617 [ 5451.814308] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5451.833709] LustreError: 188976:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5452.094948] Lustre: server umount lustre-MDT0001 complete [ 5458.999797] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 02:50:22 (1761288622) [ 5472.326442] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5481.023398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5481.560701] LustreError: 192905:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5481.619129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5486.075231] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5487.077237] LustreError: 192905:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5492.194613] LustreError: 192905:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5494.138425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5498.149462] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5500.677330] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5506.180532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5510.854628] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5518.217950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5518.276338] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5518.426825] Lustre: lustre-OST0001: new disk, initializing [ 5518.433507] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5518.526881] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5518.533448] Lustre: Skipped 2 previous similar messages [ 5523.128692] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5526.093919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5526.103873] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5526.155863] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5531.885514] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5535.830776] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5541.193180] Lustre: 194454:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 5541.207699] Lustre: 194454:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 5541.214931] Lustre: 194454:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 5541.218940] Lustre: 194454:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5541.221653] Lustre: 194454:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 5541.227823] Lustre: 194454:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5541.232688] Lustre: 194454:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [ 5541.237199] Lustre: 194454:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5541.241388] Lustre: 194454:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 5541.245671] Lustre: 194454:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5541.249308] Lustre: 194454:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 5541.253670] Lustre: 194454:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5542.896993] Lustre: DEBUG MARKER: ADD OST3 [ 5546.888111] loop: module loaded [ 5546.951931] loop0: detected capacity change from 0 to 204800000 [ 5547.137317] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5547.146328] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5547.173285] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5547.307746] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5551.479826] loop0: detected capacity change from 0 to 204800000 [ 5553.526907] Lustre: DEBUG MARKER: STOP OST3 [ 5556.115680] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5556.179214] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5556.366222] Lustre: lustre-OST0003: new disk, initializing [ 5556.375417] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5558.109105] Lustre: Failing over lustre-OST0003 [ 5558.164944] LustreError: 195653:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5558.195076] LustreError: 197266:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 5558.198939] LustreError: 197266:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5558.246581] Lustre: server umount lustre-OST0003 complete [ 5562.167813] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5570.489412] Lustre: DEBUG MARKER: START OST3 again [ 5575.953219] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5577.875146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5577.882509] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5577.904099] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 5580.230846] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5596.632936] Lustre: Failing over lustre-OST0003 [ 5596.651249] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5596.658274] Lustre: Skipped 2 previous similar messages [ 5596.658281] Lustre: lustre-OST0003: Not available for connect from 0@lo (stopping) [ 5596.658284] Lustre: Skipped 2 previous similar messages [ 5596.701093] LustreError: 199504:0:(obd_class.h:479:obd_check_dev()) Device 33 not setup [ 5596.706731] LustreError: 199504:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5596.760438] Lustre: server umount lustre-OST0003 complete [ 5601.761761] LustreError: 194362:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5601.780282] LustreError: 194362:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 12 previous similar messages [ 5612.006240] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5612.008657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5612.012125] Lustre: Skipped 3 previous similar messages [ 5612.016927] Lustre: Skipped 2 previous similar messages [ 5625.311130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5625.404848] Lustre: server umount lustre-MDT0000 complete [ 5626.923667] LustreError: 195172:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288791 with bad export cookie 15238989592436416541 [ 5626.927156] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5626.928880] LustreError: 195172:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5627.360781] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5627.366472] Lustre: Skipped 1 previous similar message [ 5631.393737] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5631.396529] Lustre: Skipped 12 previous similar messages [ 5633.169431] LustreError: 199976:0:(obd_class.h:479:obd_check_dev()) Device 29 not setup [ 5633.172197] LustreError: 199976:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5633.281794] Lustre: server umount lustre-MDT0001 complete [ 5635.236280] Lustre: server umount lustre-OST0000 complete [ 5637.327653] Lustre: server umount lustre-OST0001 complete [ 5640.447270] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 5644.018786] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5648.861951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5652.056826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5654.936104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5657.888674] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5660.965702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5660.996468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5661.105935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5661.124187] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5661.164281] Lustre: lustre-MDT0000: new disk, initializing [ 5661.210951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5661.214494] Lustre: Skipped 2 previous similar messages [ 5661.224892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5662.992858] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5668.459029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5668.488301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5668.538302] Lustre: 202561:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5668.702526] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5668.744536] Lustre: 203359:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 5668.752750] Lustre: 203359:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 2 previous similar messages [ 5668.757044] Lustre: 203359:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 5668.763952] Lustre: 203359:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 5668.770614] Lustre: 203359:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 5668.777085] Lustre: 203359:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 5668.783695] Lustre: 203359:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 5668.792096] Lustre: 203359:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 5668.796525] Lustre: 203359:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 5668.801022] Lustre: 203359:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 5668.805322] Lustre: 203359:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 5668.810647] Lustre: 203359:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 5670.546395] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5674.810858] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5676.513490] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5679.496678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5679.528781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5679.639605] Lustre: lustre-OST0000: new disk, initializing [ 5679.641471] Lustre: Skipped 1 previous similar message [ 5679.644025] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5679.646333] Lustre: Skipped 2 previous similar messages [ 5681.473224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5681.476873] Lustre: Skipped 1 previous similar message [ 5681.479364] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5681.540057] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5682.462515] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5686.392120] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5688.087319] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5688.185966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5689.843602] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5689.943479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5691.874100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5691.878597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5691.880814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5691.884983] Lustre: Skipped 1 previous similar message [ 5705.183168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5705.208430] LustreError: 205503:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5705.212724] LustreError: 205503:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 5705.321890] Lustre: server umount lustre-OST0000 complete [ 5709.792711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5709.796585] LustreError: Skipped 1 previous similar message [ 5713.744592] Lustre: server umount lustre-MDT0000 complete [ 5715.911182] LustreError: 202552:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288880 with bad export cookie 15238989592436418816 [ 5715.912408] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5715.918527] LustreError: 202552:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5716.201394] Lustre: server umount lustre-MDT0001 complete [ 5718.387550] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 02:54:42 (1761288882) [ 5728.191304] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 5733.650160] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5737.970515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5740.684507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5743.326749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5746.200781] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5749.371779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5749.404960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5749.526518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5749.548876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5749.591782] Lustre: lustre-MDT0000: new disk, initializing [ 5749.639922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5751.375432] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5756.676525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5756.708195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5756.752369] Lustre: 208749:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5756.926074] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5756.981511] Lustre: 209547:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 5756.986698] Lustre: 209547:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 5756.993382] Lustre: 209547:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 5756.999320] Lustre: 209547:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5757.003517] Lustre: 209547:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 5757.008027] Lustre: 209547:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5757.013393] Lustre: 209547:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 5757.017439] Lustre: 209547:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5757.022711] Lustre: 209547:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 5757.026141] Lustre: 209547:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5757.030508] Lustre: 209547:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 5757.034229] Lustre: 209547:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5758.653821] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5762.348667] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5763.871431] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5766.949903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5766.990244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5768.707846] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5769.812872] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5773.788751] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5775.815095] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5775.930216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5778.002904] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5778.168221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5784.036944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5784.037365] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5784.043599] Lustre: Skipped 3 previous similar messages [ 5784.049831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5784.053731] Lustre: Skipped 8 previous similar messages [ 5785.557110] Lustre: server umount lustre-OST0000 complete [ 5793.446360] Lustre: server umount lustre-MDT0000 complete [ 5794.272282] LustreError: 208760:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5794.280643] LustreError: 208760:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 10 previous similar messages [ 5795.182859] LustreError: 208739:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288959 with bad export cookie 15238989592436419740 [ 5795.184823] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5795.188664] LustreError: 208739:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5795.393415] Lustre: server umount lustre-MDT0001 complete [ 5799.535481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5799.787727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5799.790948] Lustre: Skipped 5 previous similar messages [ 5801.673096] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5805.752056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5807.710491] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5810.357292] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5812.172293] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5815.439803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5818.568759] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 5821.021909] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5835.234589] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5835.237618] LustreError: Skipped 2 previous similar messages [ 5839.602493] LustreError: 214934:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5839.606350] LustreError: 214934:0:(obd_class.h:479:obd_check_dev()) Skipped 35 previous similar messages [ 5839.684709] Lustre: server umount lustre-OST0000 complete [ 5847.667620] Lustre: server umount lustre-MDT0000 complete [ 5849.411681] LustreError: 212549:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289014 with bad export cookie 15238989592436420664 [ 5849.414431] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5849.417681] LustreError: 212549:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5849.578669] Lustre: server umount lustre-MDT0001 complete [ 5954.495909] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 5956.159703] Key type lgssc unregistered [ 5956.335800] LNet: 215905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5956.342908] LNetError: 215905:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5956.355808] LNet: Removed LNI 192.168.202.138@tcp [ 5956.840190] Key type .llcrypt unregistered [ 5956.841746] Key type ._llcrypt unregistered [ 5969.831632] Key type ._llcrypt registered [ 5969.833104] Key type .llcrypt registered [ 5969.885438] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 5977.818358] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 5978.314081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5978.325865] alg: No test for adler32 (adler32-zlib) [ 5979.210526] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 5979.323178] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 5980.927234] Key type lgssc registered [ 5981.406956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5985.313372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5988.073242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5990.736781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5993.917184] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5997.452056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5997.490486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5997.500278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5998.643100] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5998.664670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5998.710761] Lustre: lustre-MDT0000: new disk, initializing [ 5998.760576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5998.773302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6001.001669] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6006.828460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6006.870959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6006.928543] Lustre: 219208:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6006.962419] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6006.964587] Lustre: Skipped 1 previous similar message [ 6007.020296] Lustre: lustre-MDT0001: new disk, initializing [ 6007.078194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6007.099068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6007.103986] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6007.136020] Lustre: 220005:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 6007.141092] Lustre: 220005:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6007.146072] Lustre: 220005:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 6007.149992] Lustre: 220005:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 6007.154731] Lustre: 220005:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6007.157759] Lustre: 220005:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6009.222358] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6014.107429] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6016.125508] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6019.431350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6019.467046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6019.589913] Lustre: lustre-OST0000: new disk, initializing [ 6019.592710] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6019.642520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6022.205546] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6026.711368] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6027.297101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6027.303216] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6027.379416] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6028.989401] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6029.123637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6031.365571] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6031.490922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6037.476493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6037.480345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6037.482381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6037.486738] Lustre: Skipped 1 previous similar message [ 6038.646168] LustreError: 222149:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6038.744404] Lustre: server umount lustre-OST0000 complete [ 6042.596046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6042.600925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6042.605884] Lustre: Skipped 1 previous similar message [ 6042.611632] Lustre: Skipped 2 previous similar messages [ 6047.712481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6047.715420] Lustre: Skipped 1 previous similar message [ 6052.834201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6052.837629] Lustre: Skipped 1 previous similar message [ 6054.879357] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6054.937251] LustreError: 222350:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6054.940488] LustreError: 222350:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6054.978168] Lustre: server umount lustre-MDT0000 complete [ 6056.909212] LustreError: 219198:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289221 with bad export cookie 6163889408878940598 [ 6056.913759] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6056.923430] LustreError: 219198:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6057.006475] LustreError: 222552:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6057.010106] LustreError: 222552:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6057.139445] Lustre: server umount lustre-MDT0001 complete [ 6065.465269] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 6069.840699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6070.041452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6071.790517] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6075.360574] LustreError: 223725:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6075.368389] LustreError: 223725:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 6075.604239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6075.781064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6077.419230] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6078.625185] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6081.447204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6083.993477] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6087.718738] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6087.749380] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6087.827916] Lustre: lustre-OST0001: new disk, initializing [ 6087.830614] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6087.878920] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6087.882344] Lustre: Skipped 1 previous similar message [ 6090.216331] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6091.465419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6091.468965] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6091.505145] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6097.586912] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6101.847610] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6112.685793] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 03:01:16 (1761289276) [ 6115.298404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6115.300082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6115.303996] Lustre: Skipped 3 previous similar messages [ 6115.308576] Lustre: Skipped 2 previous similar messages [ 6115.725249] LustreError: 227114:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6115.728121] LustreError: 227114:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6115.783781] Lustre: server umount lustre-MDT0000 complete [ 6117.532492] LustreError: 223705:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289282 with bad export cookie 6163889408878941508 [ 6117.536128] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6117.540572] LustreError: 223705:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6117.727038] Lustre: server umount lustre-MDT0001 complete [ 6129.655634] LustreError: 227516:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6129.657860] LustreError: 227516:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 6129.682151] Lustre: server umount lustre-OST0000 complete [ 6140.476308] LustreError: 227717:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6140.478638] LustreError: 227717:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6140.590815] Lustre: server umount lustre-OST0001 complete [ 6143.257392] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 6147.452191] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 6151.877053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6155.003803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6157.978567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6161.220726] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6164.790210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6164.826348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6164.950334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6164.972189] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6165.020489] Lustre: lustre-MDT0000: new disk, initializing [ 6165.078604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6165.089397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6166.841387] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6171.789739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6171.825889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6171.870330] Lustre: 229901:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6172.014983] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6172.054993] Lustre: 230700:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 6172.058943] Lustre: 230700:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 6172.061782] Lustre: 230700:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6172.064185] Lustre: 230700:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6172.067385] Lustre: 230700:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 6172.075202] Lustre: 230700:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6172.080931] Lustre: 230700:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 6172.085298] Lustre: 230700:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6172.088606] Lustre: 230700:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6172.091628] Lustre: 230700:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6172.095037] Lustre: 230700:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6172.097596] Lustre: 230700:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6173.830293] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6177.867538] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6179.626394] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6183.246439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6183.293236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6183.450493] Lustre: lustre-OST0000: new disk, initializing [ 6183.452846] Lustre: Skipped 1 previous similar message [ 6183.455807] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6183.459482] Lustre: Skipped 2 previous similar messages [ 6185.300423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6185.304060] Lustre: Skipped 1 previous similar message [ 6185.306788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6185.370154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6186.326628] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6190.565881] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6192.504929] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6192.612302] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6194.572705] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6194.685367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6195.680741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6195.681116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6195.683989] LustreError: Skipped 1 previous similar message [ 6195.691520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6201.719375] LustreError: 232841:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6201.726161] LustreError: 232841:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6201.816298] Lustre: server umount lustre-OST0000 complete [ 6205.921926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6205.925691] Lustre: Skipped 2 previous similar messages [ 6209.776930] Lustre: server umount lustre-MDT0000 complete [ 6211.040221] LustreError: 229912:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6211.046970] LustreError: 229912:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 6211.363420] LustreError: 229894:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289375 with bad export cookie 6163889408878943622 [ 6211.367165] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6211.368272] LustreError: 229894:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6211.535100] Lustre: server umount lustre-MDT0001 complete [ 6215.406398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6215.601119] LustreError: 233719:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6215.631922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6215.634709] Lustre: Skipped 2 previous similar messages [ 6217.203544] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6220.512286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6222.329202] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6224.506025] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6225.878362] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6228.701681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6231.213668] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6233.493748] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6379.489415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6379.492108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6379.494581] LustreError: Skipped 1 previous similar message [ 6379.496772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6379.499610] Lustre: Skipped 1 previous similar message [ 6379.502836] Lustre: Skipped 5 previous similar messages [ 6382.383168] LustreError: 236006:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6382.386370] LustreError: 236006:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6382.469280] Lustre: server umount lustre-OST0000 complete [ 6384.608828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6390.058242] Lustre: server umount lustre-MDT0000 complete [ 6391.768217] LustreError: 236211:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289556 with bad export cookie 6163889408878944546 [ 6391.771909] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6391.775919] LustreError: 236211:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6391.967152] Lustre: server umount lustre-MDT0001 complete [ 6502.120419] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 6503.362916] Key type lgssc unregistered [ 6503.518790] LNet: 236978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6503.522133] LNetError: 236978:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6503.531483] LNet: Removed LNI 192.168.202.138@tcp [ 6503.886122] Key type .llcrypt unregistered [ 6503.887447] Key type ._llcrypt unregistered [ 6513.339889] Key type ._llcrypt registered [ 6513.341391] Key type .llcrypt registered [ 6513.390994] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 6519.961331] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 6520.286472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6520.298946] alg: No test for adler32 (adler32-zlib) [ 6521.169547] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 6521.273143] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 6522.871169] Key type lgssc registered [ 6523.276782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6526.733395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6529.354444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6531.689864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6534.168571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6540.233163] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 6544.265374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6544.291134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6544.297895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6545.397117] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6545.412229] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6545.455228] Lustre: lustre-MDT0000: new disk, initializing [ 6545.485667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6545.493797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6547.142801] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6552.047503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6552.079741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6552.121062] Lustre: 240976:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6552.149496] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6552.152360] Lustre: Skipped 1 previous similar message [ 6552.192532] Lustre: lustre-MDT0001: new disk, initializing [ 6552.233402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6552.248332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6552.253563] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6552.285875] Lustre: 241774:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 6552.290331] Lustre: 241774:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6552.294490] Lustre: 241774:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 6552.299077] Lustre: 241774:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 6552.307312] Lustre: 241774:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6552.311623] Lustre: 241774:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6553.805954] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6556.232688] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6558.803960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6558.832289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6558.937811] Lustre: lustre-OST0000: new disk, initializing [ 6558.940624] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6558.976407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6561.212340] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6562.940764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6562.945342] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6563.005224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6566.068626] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6566.102812] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6566.174424] Lustre: lustre-OST0001: new disk, initializing [ 6566.177518] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6566.224706] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6568.395038] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6573.062512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6573.066193] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6573.106221] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6574.019856] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6580.047158] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6597.798412] LustreError: 244502:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6597.952034] Lustre: server umount lustre-MDT0000 complete [ 6598.629150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6598.643258] Lustre: Skipped 1 previous similar message [ 6598.656720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6601.552880] LustreError: 240969:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289766 with bad export cookie 1962664600611690855 [ 6601.556487] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6601.560321] LustreError: 240969:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6601.746338] LustreError: 244703:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6601.750507] LustreError: 244703:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6602.180496] Lustre: server umount lustre-MDT0001 complete [ 6616.886756] LustreError: 244904:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6616.956757] LustreError: 244904:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6617.139864] Lustre: server umount lustre-OST0000 complete [ 6620.256165] Lustre: 238496:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761289768/real 1761289768] req@ffff8d33cc0f9f80 x1846846092809344/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761289784 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6620.291116] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6620.307612] Lustre: Skipped 2 previous similar messages [ 6621.024306] LustreError: 245106:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6621.027136] LustreError: 245106:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6621.314091] Lustre: server umount lustre-OST0001 complete [ 6635.729597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6641.880342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6642.439615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6647.265146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6652.262746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6652.872147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6664.624885] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 6670.444420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6670.458605] Lustre: lustre-MDT0000: reset Object Index mappings [ 6670.745801] LustreError: 248415:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6670.756655] LustreError: 248415:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [ 6670.799876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6673.259399] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6675.937988] LustreError: 248415:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6678.591668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6678.600353] Lustre: lustre-MDT0001: reset Object Index mappings [ 6678.856724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6681.090636] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6682.891642] Lustre: 249528:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6686.796121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6689.129388] LustreError: 249885:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6690.381681] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6694.375516] LustreError: 249885:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6695.402590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6695.560937] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6695.754473] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6695.757489] Lustre: Skipped 1 previous similar message [ 6696.906832] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6699.501912] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6704.738769] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6707.239492] Lustre: 251374:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6714.849898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6714.859469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6714.870372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6716.388125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6716.396958] Lustre: Skipped 3 previous similar messages [ 6720.787752] LustreError: 251479:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6720.791419] LustreError: 251479:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6720.805945] LustreError: 251479:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff8d34f04cf400) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6720.841823] Lustre: server umount lustre-MDT0000 complete [ 6721.507341] LustreError: 249150:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6721.518453] LustreError: 249150:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [ 6722.881022] LustreError: 248395:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289887 with bad export cookie 1962664600611697666 [ 6722.884486] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6722.888886] LustreError: 248395:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6723.123514] Lustre: server umount lustre-MDT0001 complete [ 6734.841079] LustreError: 251880:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6734.843412] LustreError: 251880:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 6734.867660] Lustre: server umount lustre-OST0000 complete [ 6747.174958] Lustre: server umount lustre-OST0001 complete [ 6757.830507] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:12:01 (1761289921) [ 6768.638464] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 6774.583733] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 6781.168362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6786.052344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6790.236092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6794.594247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6799.161114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6799.212701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6799.385815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6799.424864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6799.490689] Lustre: lustre-MDT0000: new disk, initializing [ 6799.560889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6799.570679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6802.461699] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6811.055413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6811.101085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6811.154766] Lustre: 255601:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6811.216393] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6811.219216] Lustre: Skipped 1 previous similar message [ 6811.270221] Lustre: lustre-MDT0001: new disk, initializing [ 6811.396988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6811.401322] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6811.445918] Lustre: 256400:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 6811.450519] Lustre: 256400:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 6811.454333] Lustre: 256400:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6811.458563] Lustre: 256400:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6811.463478] Lustre: 256400:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 6811.473651] Lustre: 256400:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6811.478021] Lustre: 256400:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 6811.481954] Lustre: 256400:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6811.488247] Lustre: 256400:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6811.492207] Lustre: 256400:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6811.497578] Lustre: 256400:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6811.505190] Lustre: 256400:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6813.929544] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6818.907534] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6821.440769] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6826.209945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6826.257644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6828.005822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6828.015992] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6828.100984] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6830.257027] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6835.601756] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6838.394339] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6838.546441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6841.230730] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6841.356305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6843.362871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6843.364960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6843.369778] LustreError: Skipped 1 previous similar message [ 6843.373833] Lustre: Skipped 3 previous similar messages [ 6843.382247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6843.388883] Lustre: Skipped 1 previous similar message [ 6848.485192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6848.492792] Lustre: Skipped 1 previous similar message [ 6848.707199] LustreError: 258544:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6848.713144] LustreError: 258544:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6848.835297] Lustre: server umount lustre-OST0000 complete [ 6852.577988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6852.583537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6852.591483] Lustre: Skipped 1 previous similar message [ 6852.594460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6857.675423] Lustre: server umount lustre-MDT0000 complete [ 6858.722652] LustreError: 256403:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6858.733298] LustreError: 256403:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 6860.747677] LustreError: 255595:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290025 with bad export cookie 1962664600611699766 [ 6860.748747] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6860.757536] LustreError: 255595:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6860.994650] Lustre: server umount lustre-MDT0001 complete [ 6872.769518] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 6878.975085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6879.207220] LustreError: 260123:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6879.234603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6879.237611] Lustre: Skipped 2 previous similar messages [ 6881.230587] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6885.440983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6887.690717] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6889.331987] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6893.353029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6896.593986] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6901.058383] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6901.095881] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6901.177885] Lustre: lustre-OST0001: new disk, initializing [ 6901.180525] Lustre: Skipped 1 previous similar message [ 6901.183748] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6901.186299] Lustre: Skipped 1 previous similar message [ 6902.693573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6902.698715] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6902.754420] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6904.086437] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6910.402315] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6912.925198] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6921.093180] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6921.114564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6926.822748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6926.829321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6926.836105] Lustre: Skipped 1 previous similar message [ 6926.840940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6926.843427] Lustre: Skipped 2 previous similar messages [ 6927.404531] LustreError: 263632:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6927.407452] LustreError: 263632:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6927.458858] Lustre: server umount lustre-MDT0000 complete [ 6929.376723] LustreError: 260123:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6929.383363] LustreError: 260123:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 4 previous similar messages [ 6929.664268] LustreError: 261601:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290094 with bad export cookie 1962664600611700690 [ 6929.669947] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6929.677635] LustreError: 261601:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6931.973498] Lustre: server umount lustre-MDT0001 complete [ 6943.803498] Lustre: server umount lustre-OST0000 complete [ 6956.766975] Lustre: server umount lustre-OST0001 complete [ 6965.103338] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 6970.028149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6972.200843] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6976.273863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6978.426986] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6979.903009] Lustre: 266521:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6983.290960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6986.144434] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6990.564433] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6993.574419] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 6994.544991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6994.547446] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6998.325293] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7000.718773] Lustre: 268378:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7008.993381] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 7009.027950] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 7014.765361] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:16:18 (1761290178) [ 7027.681835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7027.687415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7027.695455] Lustre: Skipped 3 previous similar messages [ 7027.700385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7033.649880] LustreError: 269083:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 7033.654514] LustreError: 269083:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 7033.734146] Lustre: server umount lustre-MDT0000 complete [ 7035.876133] LustreError: 266483:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7035.885610] LustreError: 266483:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 6 previous similar messages [ 7036.130110] LustreError: 265391:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290200 with bad export cookie 1962664600611706969 [ 7036.133366] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7036.137636] LustreError: 265391:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7042.477736] Lustre: server umount lustre-MDT0001 complete [ 7051.105224] Lustre: server umount lustre-OST0000 complete [ 7059.855684] Lustre: server umount lustre-OST0001 complete [ 7063.631755] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 7069.411462] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 7074.611688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7078.524708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7081.623582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7085.093586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7094.706714] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 7100.111663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7100.162269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7100.305613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7100.327948] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7100.378239] Lustre: lustre-MDT0000: new disk, initializing [ 7100.431961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7100.435286] Lustre: Skipped 7 previous similar messages [ 7100.448136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7102.549346] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7108.614516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7108.651549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7108.703536] Lustre: 272570:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7108.886761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7108.938805] Lustre: 273367:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 7108.944481] Lustre: 273367:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 7108.949519] Lustre: 273367:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 7108.956936] Lustre: 273367:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7108.962045] Lustre: 273367:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 7108.967386] Lustre: 273367:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7108.973090] Lustre: 273367:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 7108.977542] Lustre: 273367:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7108.981424] Lustre: 273367:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 7108.985376] Lustre: 273367:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7108.989135] Lustre: 273367:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 7108.992685] Lustre: 273367:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7111.171633] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7118.852499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7118.903793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7121.145792] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7122.735203] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7129.830417] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7129.887688] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7132.024305] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7133.240239] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7139.593843] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7141.673823] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7141.676581] Lustre: Skipped 1 previous similar message [ 7155.225635] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 03:18:39 (1761290319) [ 7157.730180] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7157.734593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7157.745715] Lustre: Skipped 8 previous similar messages [ 7157.750370] Lustre: Skipped 10 previous similar messages [ 7162.722731] LustreError: 276184:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7162.727167] LustreError: 276184:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 7162.781888] Lustre: server umount lustre-MDT0000 complete [ 7165.021145] LustreError: 272560:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290329 with bad export cookie 1962664600611714823 [ 7165.022705] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7165.026672] LustreError: 272560:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7165.234409] Lustre: server umount lustre-MDT0001 complete [ 7177.244161] Lustre: server umount lustre-OST0000 complete [ 7189.824854] Lustre: server umount lustre-OST0001 complete [ 7200.806146] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 7206.744389] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 7212.521593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7216.304887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7220.233765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7224.225520] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7228.720722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7228.765630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7228.907868] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7228.934855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7228.936801] Lustre: Skipped 4 previous similar messages [ 7228.978917] Lustre: lustre-MDT0000: new disk, initializing [ 7228.981144] Lustre: Skipped 3 previous similar messages [ 7229.056675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7229.061435] Lustre: Skipped 3 previous similar messages [ 7231.338874] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7238.451114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7238.487204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7238.535448] Lustre: 279632:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7238.781686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7238.788430] Lustre: Skipped 2 previous similar messages [ 7241.534820] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7246.814717] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7249.050630] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7253.772199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7253.813232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7255.271323] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7257.803424] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7262.403985] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7263.645914] Lustre: Mounted lustre-client [ 7264.247179] LustreError: 282136:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8d33ceeb6800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 7264.297203] Lustre: Unmounted lustre-client [ 7270.881492] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7270.881769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7270.887024] LustreError: Skipped 1 previous similar message [ 7270.893964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7270.898499] Lustre: Skipped 2 previous similar messages [ 7275.796797] Lustre: server umount lustre-OST0000 complete [ 7292.383131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7292.500397] Lustre: server umount lustre-MDT0000 complete [ 7294.433238] LustreError: 279642:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7294.442140] LustreError: 279642:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 9 previous similar messages [ 7294.633250] LustreError: 280416:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290459 with bad export cookie 1962664600611716391 [ 7294.637906] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7294.640869] LustreError: 280416:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7294.949396] Lustre: server umount lustre-MDT0001 complete [ 7300.301308] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 7302.022510] Key type lgssc unregistered [ 7317.602425] LNet: 283414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7317.606409] LNetError: 283414:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7394.406513] LNet: Removed LNI 192.168.202.138@tcp [ 7394.923104] Key type .llcrypt unregistered [ 7394.925303] Key type ._llcrypt unregistered [ 7404.797121] Key type ._llcrypt registered [ 7404.799209] Key type .llcrypt registered [ 7404.875281] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 7414.933307] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 7415.384840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7415.457156] alg: No test for adler32 (adler32-zlib) [ 7416.438027] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 7416.576768] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 7418.207188] Key type lgssc registered [ 7418.935335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7424.129086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7427.977507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7431.149092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7434.430782] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7435.397276] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 03:23:19 (1761290599) [ 7438.645403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7438.671363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7438.681856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7439.814535] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7439.841099] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7439.888909] Lustre: lustre-MDT0000: new disk, initializing [ 7439.956121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7439.976110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7441.936107] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7447.917696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7447.959826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7448.011131] Lustre: 286591:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7448.052545] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7448.055429] Lustre: Skipped 1 previous similar message [ 7448.098757] Lustre: lustre-MDT0001: new disk, initializing [ 7448.146225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7448.171807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7448.176446] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7448.219259] Lustre: 287390:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 7448.225596] Lustre: 287390:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 7448.230702] Lustre: 287390:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 7448.235949] Lustre: 287390:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 7448.242602] Lustre: 287390:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 7448.249648] Lustre: 287390:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 7450.202407] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7454.738428] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7456.615863] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7460.023199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7460.051443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7460.201196] Lustre: lustre-OST0000: new disk, initializing [ 7460.203801] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7460.208991] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 7460.264090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7462.969951] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_default_debug -1 all [ 7465.457350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7465.462638] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7465.485146] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7467.066539] Lustre: DEBUG MARKER: oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7475.683320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7475.685444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7475.688582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7475.692610] Lustre: Skipped 1 previous similar message [ 7478.198114] LustreError: 289042:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 7478.322115] Lustre: server umount lustre-OST0000 complete [ 7480.801456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7480.810924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7480.814051] Lustre: Skipped 2 previous similar messages [ 7485.920911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7485.924303] Lustre: Skipped 1 previous similar message [ 7486.287164] LustreError: 289243:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7486.289989] LustreError: 289243:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7486.353799] Lustre: server umount lustre-MDT0000 complete [ 7488.004892] LustreError: 289244:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290652 with bad export cookie 13638006033445686081 [ 7488.006912] LustreError: MGC192.168.202.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7488.011698] LustreError: 289244:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7488.086436] LustreError: 289447:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 7488.088706] LustreError: 289447:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7488.203186] Lustre: server umount lustre-MDT0001 complete [ 7492.714493] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 7494.033450] Key type lgssc unregistered [ 7494.201262] LNet: 289977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7494.205862] LNetError: 289977:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7495.271638] LNet: Removed LNI 192.168.202.138@tcp [ 7495.648891] Key type .llcrypt unregistered [ 7495.650759] Key type ._llcrypt unregistered [ 7539.551286] Key type ._llcrypt registered [ 7539.552800] Key type .llcrypt registered [ 7539.604024] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 7539.729501] Key type .llcrypt unregistered [ 7539.730969] Key type ._llcrypt unregistered [ 7573.663403] Key type ._llcrypt registered [ 7573.664774] Key type .llcrypt registered [ 7573.714280] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 7573.843524] Key type .llcrypt unregistered [ 7573.845438] Key type ._llcrypt unregistered [ 7587.763724] Key type ._llcrypt registered [ 7587.765277] Key type .llcrypt registered [ 7587.826580] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 7587.971567] Key type .llcrypt unregistered [ 7587.973403] Key type ._llcrypt unregistered [ 7625.682754] Key type ._llcrypt registered [ 7625.684077] Key type .llcrypt registered [ 7625.740139] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 7625.868738] Key type .llcrypt unregistered [ 7625.870526] Key type ._llcrypt unregistered [ 7639.220946] Key type ._llcrypt registered [ 7639.222323] Key type .llcrypt registered [ 7639.271604] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 7639.400722] Key type .llcrypt unregistered [ 7639.402444] Key type ._llcrypt unregistered [ 7653.158355] Key type ._llcrypt registered [ 7653.160018] Key type .llcrypt registered [ 7653.215374] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 7653.342322] Key type .llcrypt unregistered [ 7653.343829] Key type ._llcrypt unregistered [ 7689.853620] Key type ._llcrypt registered [ 7689.855463] Key type .llcrypt registered [ 7689.911607] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 7690.034677] Key type .llcrypt unregistered [ 7690.036574] Key type ._llcrypt unregistered [ 7703.701177] Key type ._llcrypt registered [ 7703.702743] Key type .llcrypt registered [ 7703.748212] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 7703.892046] Key type .llcrypt unregistered [ 7703.893775] Key type ._llcrypt unregistered [ 7718.309390] Key type ._llcrypt registered [ 7718.310820] Key type .llcrypt registered [ 7718.360708] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 7718.487534] Key type .llcrypt unregistered [ 7718.489115] Key type ._llcrypt unregistered [ 7743.524308] Key type ._llcrypt registered [ 7743.525418] Key type .llcrypt registered [ 7743.579776] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing set_hostid [ 7751.656353] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing load_modules_local [ 7752.057632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7752.103331] alg: No test for adler32 (adler32-zlib) [ 7753.004209] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 7753.112625] LNet: Added LNI 192.168.202.138@tcp [8/256/0/180] [ 7754.711188] Key type lgssc registered [ 7755.226522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7759.523446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7763.079653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7765.782972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7769.150155] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7769.936257] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7620 sec ========== 03:28:54 (1761290934) [ 7770.772899] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:28:54 (1761290934) === [ 7773.272246] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:28:56 (1761290936) === [ 7818.724788] Lustre: DEBUG MARKER: oleg238-server.virtnet: executing unload_modules_local [ 7824.461977] Key type lgssc unregistered [ 7824.806355] LNet: 308778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7824.827224] LNetError: 308778:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7824.913191] LNet: Removed LNI 192.168.202.138@tcp [ 7826.671277] Key type .llcrypt unregistered [ 7826.673069] Key type ._llcrypt unregistered