[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 363837680 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.522874] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.526072] pid_max: default: 32768 minimum: 301 [ 0.527887] Security Framework initialized [ 0.529454] SELinux: Initializing. [ 0.530662] SELinux: Starting in permissive mode [ 0.532398] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.537033] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.539948] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.542738] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.547394] Initializing cgroup subsys memory [ 0.549351] Initializing cgroup subsys devices [ 0.551396] Initializing cgroup subsys freezer [ 0.553615] Initializing cgroup subsys net_cls [ 0.555371] Initializing cgroup subsys blkio [ 0.557257] Initializing cgroup subsys perf_event [ 0.559138] Initializing cgroup subsys hugetlb [ 0.561102] Initializing cgroup subsys pids [ 0.562566] Initializing cgroup subsys net_prio [ 0.564295] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.567746] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.569746] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.572224] tlb_flushall_shift: 6 [ 0.573498] FEATURE SPEC_CTRL Present [ 0.575174] FEATURE IBPB_SUPPORT Present [ 0.576423] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.578567] Spectre V2 : Vulnerable [ 0.579670] Speculative Store Bypass: Vulnerable [ 0.582354] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.592291] ACPI: Core revision 20130517 [ 0.595575] ACPI: All ACPI Tables successfully acquired [ 0.598403] ftrace: allocating 30294 entries in 119 pages [ 0.660636] Enabling x2apic [ 0.661545] Enabled x2apic [ 0.662887] Switched APIC routing to physical x2apic. [ 0.666755] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.668632] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.671902] TSC deadline timer enabled [ 0.672218] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.678067] ... version: 2 [ 0.679503] ... bit width: 48 [ 0.683965] ... generic registers: 4 [ 0.685353] ... value mask: 0000ffffffffffff [ 0.692931] ... max period: 00007fffffffffff [ 0.694552] ... fixed-purpose events: 3 [ 0.695776] ... event mask: 000000070000000f [ 0.701284] KVM setup paravirtual spinlock [ 0.710402] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.722790] KVM setup async PF for cpu 1 [ 0.724148] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.734756] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.707516] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.742779] KVM setup async PF for cpu 2 [ 0.750791] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.752414] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.756557] Brought up 4 CPUs [ 0.756754] KVM setup async PF for cpu 3 [ 0.756761] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.762137] smpboot: Max logical packages: 1 [ 0.763602] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.779390] devtmpfs: initialized [ 0.780812] x86/mm: Memory block size: 128MB [ 0.785503] EVM: security.selinux [ 0.786887] EVM: security.ima [ 0.788127] EVM: security.capability [ 0.797653] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.800311] NET: Registered protocol family 16 [ 0.802387] cpuidle: using governor haltpoll [ 0.804670] ACPI: bus type PCI registered [ 0.806078] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.808736] PCI: Using configuration type 1 for base access [ 0.810462] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.824473] ACPI: Added _OSI(Module Device) [ 0.827478] ACPI: Added _OSI(Processor Device) [ 0.829069] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.831382] ACPI: Added _OSI(Processor Aggregator Device) [ 0.833365] ACPI: Added _OSI(Linux-Dell-Video) [ 0.836025] ACPI: EC: Look up EC in DSDT [ 0.839090] ACPI: Interpreter enabled [ 0.840611] ACPI: (supports S0 S3 S4 S5) [ 0.842219] ACPI: Using IOAPIC for interrupt routing [ 0.844225] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.847624] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.855738] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.858150] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.860428] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.862696] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.867344] acpiphp: Slot [2] registered [ 0.868863] acpiphp: Slot [3] registered [ 0.870373] acpiphp: Slot [4] registered [ 0.871886] acpiphp: Slot [5] registered [ 0.873366] acpiphp: Slot [6] registered [ 0.874930] acpiphp: Slot [7] registered [ 0.876749] acpiphp: Slot [8] registered [ 0.878276] acpiphp: Slot [9] registered [ 0.879936] acpiphp: Slot [10] registered [ 0.891526] acpiphp: Slot [11] registered [ 0.893031] acpiphp: Slot [12] registered [ 0.894585] acpiphp: Slot [13] registered [ 0.896190] acpiphp: Slot [14] registered [ 0.897881] acpiphp: Slot [15] registered [ 0.899310] acpiphp: Slot [16] registered [ 0.900710] acpiphp: Slot [17] registered [ 0.902110] acpiphp: Slot [18] registered [ 0.903815] acpiphp: Slot [19] registered [ 0.905220] acpiphp: Slot [20] registered [ 0.906989] acpiphp: Slot [21] registered [ 0.908400] acpiphp: Slot [22] registered [ 0.909728] acpiphp: Slot [23] registered [ 0.911012] acpiphp: Slot [24] registered [ 0.912374] acpiphp: Slot [25] registered [ 0.914076] acpiphp: Slot [26] registered [ 0.915510] acpiphp: Slot [27] registered [ 0.917113] acpiphp: Slot [28] registered [ 0.918406] acpiphp: Slot [29] registered [ 0.920163] acpiphp: Slot [30] registered [ 0.921725] acpiphp: Slot [31] registered [ 0.923488] PCI host bridge to bus 0000:00 [ 0.925086] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.927527] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.929968] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.932763] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.935425] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.938459] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.940459] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.941756] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.942655] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.949692] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.952967] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.955886] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.958181] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.960517] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.963450] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.964129] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.966818] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.970000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.972206] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.985590] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.992979] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.994279] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.998518] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 1.007988] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 1.026766] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 1.038362] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 1.043363] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 1.048923] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 1.067053] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 1.083099] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 1.093469] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 1.112539] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 1.134031] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 1.143366] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.152241] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.157455] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.177925] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 1.183728] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.199830] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.215486] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.237527] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 1.248601] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.256412] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.259375] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.282263] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 1.286694] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 1.289002] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 1.293301] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 1.295416] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 1.297524] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 1.301367] vgaarb: loaded [ 1.302492] SCSI subsystem initialized [ 1.303712] ACPI: bus type USB registered [ 1.305264] usbcore: registered new interface driver usbfs [ 1.307106] usbcore: registered new interface driver hub [ 1.310602] usbcore: registered new device driver usb [ 1.314441] PCI: Using ACPI for IRQ routing [ 1.316070] PCI: pci_cache_line_size set to 64 bytes [ 1.316341] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 1.316346] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 1.316348] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 1.316676] NetLabel: Initializing [ 1.317800] NetLabel: domain hash size = 128 [ 1.319441] NetLabel: protocols = UNLABELED CIPSOv4 [ 1.321956] NetLabel: unlabeled traffic allowed by default [ 1.324298] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.326239] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.334409] amd_nb: Cannot enumerate AMD northbridges [ 1.345424] Switched to clocksource kvm-clock [ 1.368967] pnp: PnP ACPI init [ 1.370714] ACPI: bus type PNP registered [ 1.371890] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 1.371993] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 1.372042] pnp 00:02: [dma 2] [ 1.372080] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 1.372165] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.372260] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.372350] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.372714] pnp: PnP ACPI: found 6 devices [ 1.374755] ACPI: bus type PNP unregistered [ 1.394036] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.394042] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.394044] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.394047] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.394049] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 1.394109] NET: Registered protocol family 2 [ 1.397834] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 1.401049] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.404621] TCP: Hash tables configured (established 32768 bind 32768) [ 1.407182] TCP: reno registered [ 1.408397] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.410671] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.413355] NET: Registered protocol family 1 [ 1.419270] RPC: Registered named UNIX socket transport module. [ 1.421323] RPC: Registered udp transport module. [ 1.422636] RPC: Registered tcp transport module. [ 1.424149] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.426294] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.428230] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.430201] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.432082] PCI: CLS 0 bytes, default 64 [ 1.432226] Unpacking initramfs... [ 4.035437] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 4.040660] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.042955] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 4.046182] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 4.048985] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 4.050836] RAPL PMU: hw unit of domain package 2^-0 Joules [ 4.052720] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 4.059425] cryptomgr_test (52) used greatest stack depth: 14480 bytes left [ 4.060919] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 4.060956] Initialise system trusted keyring [ 4.106514] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 4.108485] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 4.118633] zpool: loaded [ 4.119453] zbud: loaded [ 4.120644] VFS: Disk quotas dquot_6.6.0 [ 4.121937] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 4.126095] NFS: Registering the id_resolver key type [ 4.128011] Key type id_resolver registered [ 4.130024] Key type id_legacy registered [ 4.131288] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 4.133985] Key type big_key registered [ 4.135284] SELinux: Registering netfilter hooks [ 4.141739] cryptomgr_test (58) used greatest stack depth: 14048 bytes left [ 4.147174] cryptomgr_test (63) used greatest stack depth: 13984 bytes left [ 4.158880] NET: Registered protocol family 38 [ 4.161493] Key type asymmetric registered [ 4.163289] Asymmetric key parser 'x509' registered [ 4.165033] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 4.168743] io scheduler noop registered [ 4.169949] io scheduler deadline registered (default) [ 4.171700] io scheduler cfq registered [ 4.175503] io scheduler mq-deadline registered [ 4.177101] io scheduler kyber registered [ 4.192228] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 4.194078] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 4.197043] intel_idle: does not run on family 6 model 62 [ 4.197233] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.199752] ACPI: Power Button [PWRF] [ 4.201547] GHES: HEST is not enabled! [ 4.285921] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 4.380617] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 4.568469] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 4.731047] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 5.057667] tsc: Refined TSC clocksource calibration: 2399.956 MHz [ 5.238762] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.300699] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 5.341096] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 5.363362] Non-volatile memory driver v1.3 [ 5.374761] Linux agpgart interface v0.103 [ 5.380875] crash memory driver: version 1.1 [ 5.388888] nbd: registered device at major 43 [ 5.417351] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 5.417401] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 5.417442] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 5.417488] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 5.419620] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 5.459540] virtio_blk virtio1: [vda] 113176 512-byte logical blocks (57.9 MB/55.2 MiB) [ 5.511714] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 5.511767] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 5.511809] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 5.511864] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 5.511912] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 5.543210] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.569649] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 5.569710] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 5.569772] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 5.569814] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 5.569865] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 5.607888] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.636349] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 5.636404] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 5.636446] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 5.636483] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 5.636520] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 5.682017] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.708220] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 5.708271] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 5.708307] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 5.708348] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 5.708379] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 5.758953] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.788614] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 5.788663] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 5.788707] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 5.788747] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 5.788797] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 5.797379] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.805142] rdac: device handler registered [ 5.807340] hp_sw: device handler registered [ 5.808668] emc: device handler registered [ 5.810363] libphy: Fixed MDIO Bus: probed [ 5.814858] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 5.816386] ehci-pci: EHCI PCI platform driver [ 5.817515] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 5.819588] ohci-pci: OHCI PCI platform driver [ 5.820712] uhci_hcd: USB Universal Host Controller Interface driver [ 5.823082] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.829607] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.832845] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.838197] mousedev: PS/2 mouse device common for all mice [ 5.841946] rtc_cmos 00:05: RTC can wake from S4 [ 5.844257] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 5.847548] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.856122] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.878282] hidraw: raw HID events driver (C) Jiri Kosina [ 5.880521] usbcore: registered new interface driver usbhid [ 5.882471] usbhid: USB HID core driver [ 5.883951] drop_monitor: Initializing network drop monitor service [ 5.886170] Netfilter messages via NETLINK v0.30. [ 5.888296] TCP: cubic registered [ 5.889500] Initializing XFRM netlink socket [ 5.891579] NET: Registered protocol family 10 [ 5.896935] NET: Registered protocol family 17 [ 5.898918] Key type dns_resolver registered [ 5.901589] mce: Using 10 MCE banks [ 5.903928] Loading compiled-in X.509 certificates [ 5.908687] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 5.912532] registered taskstats version 1 [ 5.919940] modprobe (72) used greatest stack depth: 13456 bytes left [ 5.933759] Key type trusted registered [ 5.938758] modprobe (78) used greatest stack depth: 13376 bytes left [ 5.956691] Key type encrypted registered [ 5.958050] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 5.964162] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 5.972769] rtc_cmos 00:05: setting system clock to 2024-12-15 09:29:12 UTC (1734254952) [ 5.976747] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 5.988639] Write protecting the kernel read-only data: 12288k [ 5.996037] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 6.014153] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 6.052328] random: systemd: uninitialized urandom read (16 bytes read) [ 6.059833] random: systemd: uninitialized urandom read (16 bytes read) [ 6.065076] random: systemd: uninitialized urandom read (16 bytes read) [ 6.071476] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 6.088900] systemd[1]: Detected virtualization kvm. [ 6.094649] systemd[1]: Detected architecture x86-64. [ 6.103205] systemd[1]: Running in initial RAM disk. [ 6.122060] systemd[1]: No hostname configured. [ 6.123638] systemd[1]: Set hostname to . [ 6.125561] random: systemd: uninitialized urandom read (16 bytes read) [ 6.129834] systemd[1]: Initializing machine ID from random generator. [ 6.211208] dracut-rootfs-g (86) used greatest stack depth: 13264 bytes left [ 6.217128] random: systemd: uninitialized urandom read (16 bytes read) [ 6.223003] random: systemd: uninitialized urandom read (16 bytes read) [ 6.225336] random: systemd: uninitialized urandom read (16 bytes read) [ 6.228012] random: systemd: uninitialized urandom read (16 bytes read) [ 6.234698] random: systemd: uninitialized urandom read (16 bytes read) [ 6.238300] random: systemd: uninitialized urandom read (16 bytes read) [ 6.268806] systemd[1]: Reached target Local File Systems. [ 6.282186] systemd[1]: Reached target Timers. [ 6.297751] systemd[1]: Reached target Swap. [ 6.306399] systemd[1]: Created slice Root Slice. [ 6.314936] systemd[1]: Listening on udev Kernel Socket. [ 6.322756] systemd[1]: Created slice System Slice. [ 6.329589] systemd[1]: Listening on udev Control Socket. [ 6.337666] systemd[1]: Listening on Journal Socket. [ 6.347567] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 6.359925] systemd[1]: Starting Setup Virtual Console... [ 6.367791] systemd[1]: Starting Load Kernel Modules... [ 6.371989] systemd[1]: Reached target Sockets. [ 6.379624] systemd[1]: Starting dracut cmdline hook... [ 6.391884] systemd[1]: Reached target Slices. [ 6.403180] systemd[1]: Starting Journal Service... [ 6.418393] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 6.456242] systemd[1]: Started Setup Virtual Console. [ 6.473802] systemd[1]: Started Load Kernel Modules. [ 6.504921] systemd[1]: Starting Apply Kernel Variables... [ 6.554739] systemd[1]: Starting Create Static Device Nodes in /dev... [ 6.575893] systemd[1]: Started Journal Service. [ 6.812386] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 8.140698] random: fast init done [ 8.366417] libata version 3.00 loaded. [ 8.451101] ip (304) used greatest stack depth: 13080 bytes left [ 8.607725] ip (335) used greatest stack depth: 12592 bytes left [ 8.692895] ip (336) used greatest stack depth: 12336 bytes left [ 8.966307] ata_piix 0000:00:01.1: version 2.13 [ 8.984366] scsi host0: ata_piix [ 9.006473] scsi host1: ata_piix [ 9.008134] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.011343] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.305792] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 14.075406] systemd-journald[107]: Received SIGTERM from PID 1 (systemd). [ 14.813004] SELinux: Disabled at runtime. [ 14.814621] SELinux: Unregistering netfilter hooks [ 15.092539] ip_tables: (C) 2000-2006 Netfilter Core Team [ 15.106963] systemd[1]: Inserted module 'ip_tables' [ 15.388934] systemd-sysv-ge (566) used greatest stack depth: 11856 bytes left [ 16.913151] systemd-journald[573]: Received request to flush runtime journal from PID 1 [ 17.494724] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 17.952914] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.985782] cryptd: max_cpu_qlen set to 1000 [ 18.103206] AVX version of gcm_enc/dec engaged. [ 18.104947] AES CTR mode by8 optimization enabled [ 18.156612] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 18.197804] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 18.203131] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 18.203999] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 18.420809] EDAC MC: Ver: 3.0.0 [ 18.431232] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 18.431261] EDAC sbridge: Ver: 1.1.2 [ 26.732867] mount.nfs (785) used greatest stack depth: 10704 bytes left [ 31.382362] hrtimer: interrupt took 2422370 ns [ 50.281703] device-mapper: uevent: version 1.0.3 [ 50.286896] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 59.626625] libcfs: loading out-of-tree module taints kernel. [ 59.628812] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 59.710139] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 67.817556] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 68.133854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 68.140698] alg: No test for adler32 (adler32-zlib) [ 69.116479] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 69.394943] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 69.399592] LNet: Accept secure, port 988 [ 70.993863] Key type lgssc registered [ 71.608851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 77.325311] icp: module license 'CDDL' taints kernel. [ 77.327539] Disabling lock debugging due to kernel taint [ 79.936464] ZFS: Loaded module v2.1.5-1, ZFS pool version 5000, ZFS filesystem version 5 [ 85.470440] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 95.613749] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 100.727527] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 100.735133] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 105.839735] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 105.865651] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 113.719115] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 120.242206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 120.274081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 120.286302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 121.453893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 121.477657] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 121.518704] Lustre: lustre-MDT0000: new disk, initializing [ 121.568055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 121.586878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 121.656507] mount.lustre (7250) used greatest stack depth: 10224 bytes left [ 123.131795] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 130.799636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 130.854608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 130.905918] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 130.917342] Lustre: 8412:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 130.947001] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 130.950592] Lustre: Skipped 1 previous similar message [ 131.011759] Lustre: lustre-MDT0001: new disk, initializing [ 131.081003] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 131.109623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 131.115910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 133.092832] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 136.139312] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 141.385959] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 141.397292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 141.444814] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 141.454512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 141.586895] Lustre: 9616:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 141.606303] Lustre: lustre-OST0000: new disk, initializing [ 141.623875] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 141.675664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 144.076791] random: crng init done [ 144.177579] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 148.180497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 148.186763] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 148.235871] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 151.416806] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 151.425133] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 151.457183] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 151.467426] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 151.527960] Lustre: 10890:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 151.552107] Lustre: lustre-OST0001: new disk, initializing [ 151.560009] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 151.597224] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 155.134012] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 157.670652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 157.678854] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 157.696804] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 161.695282] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 168.390472] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 175.260014] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing check_logdir /tmp/testlogs/ [ 176.425759] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing yml_node [ 177.707924] Lustre: DEBUG MARKER: Client: 2.16.50.90 [ 178.601322] Lustre: DEBUG MARKER: MDS: 2.16.50.90 [ 179.397809] Lustre: DEBUG MARKER: OSS: 2.16.50.90 [ 179.838913] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Dec 15 04:32:06 EST 2024 [ 185.167788] Lustre: DEBUG MARKER: excepting tests: 55 63 67 110 [ 185.629392] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 111 114 [ 197.740020] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 197.740771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 197.750009] Lustre: Skipped 2 previous similar messages [ 202.746754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 202.749051] Lustre: Skipped 6 previous similar messages [ 202.937897] Lustre: server umount lustre-MDT0000 complete [ 204.211466] LustreError: 8424:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255150 with bad export cookie 15628271989328205181 [ 204.214477] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 204.222417] LustreError: 8424:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 207.755758] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 207.755998] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 207.763033] Lustre: Skipped 2 previous similar messages [ 209.758988] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 209.761922] Lustre: Skipped 2 previous similar messages [ 210.351703] Lustre: server umount lustre-MDT0001 complete [ 211.711971] Lustre: server umount lustre-OST0000 complete [ 213.207271] Lustre: server umount lustre-OST0001 complete [ 215.187768] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 217.578795] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 221.070128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 223.665572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 226.323745] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 226.337586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 228.896956] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 228.907019] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 232.016383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 232.063418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 232.191719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 232.213984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 232.273496] Lustre: lustre-MDT0000: new disk, initializing [ 232.315075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 232.326129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 233.850482] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 239.693739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 239.728549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 239.785680] Lustre: 18629:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 239.949883] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 239.975539] mount.lustre (18629) used greatest stack depth: 9616 bytes left [ 241.591093] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 246.234083] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 247.886803] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 250.783554] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 250.793881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 250.819750] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 250.826057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 250.937536] Lustre: 19826:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 250.955106] Lustre: lustre-OST0000: new disk, initializing [ 250.958604] Lustre: Skipped 1 previous similar message [ 250.963787] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 250.968080] Lustre: Skipped 2 previous similar messages [ 251.007881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 251.010176] Lustre: Skipped 1 previous similar message [ 252.029624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 252.032547] Lustre: Skipped 1 previous similar message [ 252.037683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 252.098812] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 253.097030] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 256.904009] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 258.436062] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 258.492062] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 259.522720] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 259.584313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 262.076867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 262.082923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 262.090083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 262.325293] Lustre: server umount lustre-OST0000 complete [ 264.941613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 264.953486] LustreError: Skipped 1 previous similar message [ 264.957702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 264.978335] Lustre: Skipped 1 previous similar message [ 270.273778] Lustre: server umount lustre-MDT0000 complete [ 271.991126] LustreError: 19850:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255218 with bad export cookie 15628271989328206714 [ 271.993427] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 272.000333] LustreError: 19850:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 272.139390] Lustre: server umount lustre-MDT0001 complete [ 273.626694] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 04:33:40 (1734255220) [ 274.741196] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 275.654339] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 04:33:42 (1734255222) [ 278.739633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 278.962524] LustreError: 22641:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 278.972086] LustreError: 22641:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 5 previous similar messages [ 279.000764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 280.074886] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 283.120521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 283.181088] Lustre: 23502:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 284.362324] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 286.209364] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 287.387037] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 289.577700] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 289.582752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 291.443509] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 293.309179] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 312.794856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 312.797807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.802232] Lustre: Skipped 1 previous similar message [ 312.804521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 312.807396] Lustre: Skipped 4 previous similar messages [ 316.732504] Lustre: server umount lustre-OST0000 complete [ 318.314915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 318.318339] LustreError: Skipped 1 previous similar message [ 323.735830] Lustre: server umount lustre-MDT0000 complete [ 323.802739] LustreError: 22641:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.807336] LustreError: 22641:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 324.636069] LustreError: 22622:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255271 with bad export cookie 15628271989328207596 [ 324.637542] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 324.645064] LustreError: 22622:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 324.733051] Lustre: server umount lustre-MDT0001 complete [ 327.723535] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 328.281775] Key type lgssc unregistered [ 328.361562] LNet: 26496:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.368678] LNet: Removed LNI 192.168.204.148@tcp [ 335.641765] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 335.772545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 335.810042] alg: No test for adler32 (adler32-zlib) [ 336.630617] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 336.696773] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 336.699216] LNet: Accept secure, port 988 [ 338.237406] Key type lgssc registered [ 338.521420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 340.890162] loop: module loaded [ 341.031001] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 343.045657] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 343.049651] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 346.730555] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 346.750032] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 346.757383] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 347.846922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 347.857140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 347.892511] Lustre: lustre-MDT0000: new disk, initializing [ 347.921731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 347.928879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 348.871196] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 354.708925] LDISKFS-fs (dm-5): file extents enabled, maximum tree depth=5 [ 354.712591] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 354.733057] LDISKFS-fs (dm-5): file extents enabled, maximum tree depth=5 [ 354.736344] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 354.828941] Lustre: 30196:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 354.844714] Lustre: lustre-OST07c6: new disk, initializing [ 354.847152] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 354.849498] Lustre: Skipped 1 previous similar message [ 354.873305] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 355.495916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 355.501265] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 355.511565] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 356.460210] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 440.279454] Lustre: server umount lustre-OST07c6 complete [ 452.820082] Lustre: server umount lustre-MDT0000 complete [ 457.275309] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 457.867482] Key type lgssc unregistered [ 457.967798] LNet: 828:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 458.973519] LNet: Removed LNI 192.168.204.148@tcp [ 464.106782] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 464.269352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 464.274838] alg: No test for adler32 (adler32-zlib) [ 465.088287] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 465.139040] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 465.140840] LNet: Accept secure, port 988 [ 466.672392] Key type lgssc registered [ 466.885079] Lustre: Echo OBD driver; http://www.lustre.org/ [ 468.919954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 468.927137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 470.070524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.888650] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 473.102669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 473.141297] Lustre: 3030:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 473.210783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 473.228056] mount.lustre (3030) used greatest stack depth: 9520 bytes left [ 474.078886] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 475.448261] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 476.421586] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 478.452367] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 478.458451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 478.546209] Lustre: 4141:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 478.596769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 479.596844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 480.155458] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 481.611044] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 484.393700] LustreError: 4882:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 487.612657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 487.612902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 487.613528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 487.623035] Lustre: Skipped 1 previous similar message [ 492.623813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 492.627374] Lustre: Skipped 2 previous similar messages [ 492.714188] Lustre: server umount lustre-OST0000 complete [ 497.627220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 497.627872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 497.634855] Lustre: Skipped 1 previous similar message [ 499.955906] Lustre: server umount lustre-MDT0000 complete [ 500.971310] LustreError: 2154:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255447 with bad export cookie 11662239287511277959 [ 500.972437] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 500.978540] LustreError: 2154:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 501.088118] Lustre: server umount lustre-MDT0001 complete [ 504.035719] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 504.549271] Key type lgssc unregistered [ 504.629549] LNet: 6289:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 505.634482] LNet: Removed LNI 192.168.204.148@tcp [ 510.495853] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 510.607794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 510.612859] alg: No test for adler32 (adler32-zlib) [ 511.417768] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 511.461834] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 511.463610] LNet: Accept secure, port 988 [ 512.986381] Key type lgssc registered [ 513.167353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 515.136096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 515.142695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 516.300407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 517.084827] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 519.237175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 519.267020] Lustre: 8471:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 519.349734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 520.136673] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 521.522168] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 522.417880] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 524.191711] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 524.194958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 524.254524] Lustre: 9584:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 524.289660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 525.591418] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 526.291126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 526.894646] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 536.314763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 536.317785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.322765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 539.306900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 539.307210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 539.312122] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 542.083164] Lustre: server umount lustre-OST0000 complete [ 544.314838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.318841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 544.320589] Lustre: Skipped 2 previous similar messages [ 549.090570] Lustre: server umount lustre-MDT0000 complete [ 549.322664] LustreError: 7610:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.329632] LustreError: 7610:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 3 previous similar messages [ 550.092970] LustreError: 10659:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255496 with bad export cookie 2738666764216676718 [ 550.094470] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 550.099315] LustreError: 10659:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 550.192526] Lustre: server umount lustre-MDT0001 complete [ 553.127907] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 553.685230] Key type lgssc unregistered [ 553.789747] LNet: 11462:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 554.796508] LNet: Removed LNI 192.168.204.148@tcp [ 561.238049] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 561.359325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 561.396705] alg: No test for adler32 (adler32-zlib) [ 562.216945] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 562.265171] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 562.266719] LNet: Accept secure, port 988 [ 563.795387] Key type lgssc registered [ 563.994565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 565.885849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 565.893756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 567.059563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 567.914732] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 570.342487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 570.390218] Lustre: 13837:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 570.484863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 571.341703] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 572.744669] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 573.630555] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 575.485597] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 575.492866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 575.566576] Lustre: 14948:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 575.609080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 576.610770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 577.090289] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 578.444894] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 589.572188] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.48@tcp (stopping) [ 589.626938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.631815] Lustre: Skipped 2 previous similar messages [ 592.420323] Lustre: server umount lustre-MDT0000 complete [ 593.372037] LustreError: 12959:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255539 with bad export cookie 11307699581004319953 [ 593.373549] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 593.379675] LustreError: 12959:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 594.575774] LustreError: 13846:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.582837] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.48@tcp (stopping) [ 594.582839] Lustre: Skipped 3 previous similar messages [ 594.590174] LustreError: 13846:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 594.635219] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 598.642817] LustreError: 12974:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 598.642882] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 598.642884] Lustre: Skipped 1 previous similar message [ 598.654882] LustreError: 12974:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 599.472638] Lustre: server umount lustre-MDT0001 complete [ 609.982349] LustreError: 12960:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.48@tcp arrived at 1734255556 with bad export cookie 11307699581004320828 [ 609.987975] LustreError: 12960:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 616.565184] Lustre: server umount lustre-OST0000 complete [ 620.893481] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 621.445925] Key type lgssc unregistered [ 621.539597] LNet: 17237:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 622.548599] LNet: Removed LNI 192.168.204.148@tcp [ 628.359551] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 628.456077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 628.460209] alg: No test for adler32 (adler32-zlib) [ 629.263723] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 629.303172] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 629.304893] LNet: Accept secure, port 988 [ 630.827362] Key type lgssc registered [ 630.998996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 632.728375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 632.732973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.852947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 634.573391] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 636.639585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 636.664526] Lustre: 19610:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 636.732037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 637.439877] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 638.654713] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 639.432056] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 641.105677] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 641.110348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 641.169802] Lustre: 20721:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 641.203735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 642.398544] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 643.770687] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 644.203836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 652.220370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 652.221439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 652.224201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 654.903590] Lustre: lustre-OST0000: Not available for connect from 192.168.204.48@tcp (stopping) [ 654.908080] Lustre: Skipped 1 previous similar message [ 657.228558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 657.232286] Lustre: Skipped 1 previous similar message [ 657.837813] Lustre: server umount lustre-OST0000 complete [ 671.771322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 671.776591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 671.782184] Lustre: Skipped 1 previous similar message [ 671.784634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 676.274680] Lustre: server umount lustre-MDT0000 complete [ 677.259473] LustreError: 19618:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 677.264547] LustreError: 19618:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 3 previous similar messages [ 677.455898] LustreError: 18734:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255623 with bad export cookie 933453643850731005 [ 677.457205] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 677.464421] LustreError: 18734:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 683.608759] Lustre: server umount lustre-MDT0001 complete [ 687.893651] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 688.882301] Key type lgssc unregistered [ 688.996474] LNet: 22813:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.004698] LNet: Removed LNI 192.168.204.148@tcp [ 700.136546] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 700.267719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 700.300407] alg: No test for adler32 (adler32-zlib) [ 701.119117] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 701.171742] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 701.174529] LNet: Accept secure, port 988 [ 702.715918] Key type lgssc registered [ 702.941896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 703.721121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 703.726807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 704.878270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 705.182753] Lustre: Failing over lustre-MDT0000 [ 705.185298] LustreError: 24283:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 705.190073] LustreError: 24283:0:(lod_sub_object.c:914:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 705.193771] LustreError: 24283:0:(lod_dev.c:506:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -5 [ 705.300426] Lustre: server umount lustre-MDT0000 complete [ 706.818620] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 04:40:53 (1734255653) [ 708.548756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 708.677347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 709.491280] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 711.722207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 711.755155] Lustre: 25959:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 711.828219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 712.630190] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 713.964438] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 714.772978] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 716.601527] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 716.604837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 716.672829] Lustre: 27072:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 716.712433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 718.011337] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 718.713004] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 719.406856] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 727.207028] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 731.723058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 731.723459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 731.729994] Lustre: Skipped 1 previous similar message [ 734.727653] Lustre: server umount lustre-MDT0000 complete [ 735.911351] LustreError: 28000:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255682 with bad export cookie 3153050845363203523 [ 735.912602] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 735.922588] LustreError: 28000:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 736.047528] Lustre: server umount lustre-MDT0001 complete [ 737.107093] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 738.618372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 741.087383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 745.732664] LustreError: 23822:0:(client.c:1310:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8800731f0000 x1818498849333120/t0(0) o250->MGC192.168.204.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 745.879059] LustreError: 27088:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 745.888299] LustreError: 27088:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 3 previous similar messages [ 745.909773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 745.956580] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 747.060634] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 750.409876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 750.482646] Lustre: 30273:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 750.579414] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 750.586561] Lustre: Skipped 1 previous similar message [ 751.625834] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 752.040904] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 752.620346] Lustre: 23826:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734255683/real 1734255683] req@ffff88009d2c6300 x1818498849332736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1734255699 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 754.080182] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 755.208958] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 756.637804] Lustre: 23826:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734255687/real 1734255687] req@ffff88009d2c7b80 x1818498849332864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1734255703 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 760.618849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 760.622453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 760.629029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 760.632977] Lustre: Skipped 2 previous similar messages [ 761.627359] Lustre: 23823:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734255692/real 1734255692] req@ffff8800731f0e00 x1818498849333248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1734255708 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 761.638851] Lustre: 23823:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 761.643635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 761.645740] Lustre: Skipped 2 previous similar messages [ 763.355432] Lustre: server umount lustre-MDT0000 complete [ 764.902516] LustreError: 25078:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255711 with bad export cookie 3153050845363205840 [ 764.904389] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 764.911340] LustreError: 25078:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 765.060696] Lustre: server umount lustre-MDT0001 complete [ 767.949317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 772.021113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 774.653947] LustreError: 23822:0:(client.c:1310:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88012fdb9180 x1818498849352832/t0(0) o250->MGC192.168.204.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 774.833134] LustreError: 30282:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 774.876353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 774.879617] Lustre: Skipped 1 previous similar message [ 774.915790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000bd0:3 to 0x280000bd0:33) [ 776.325794] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 779.403994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 779.466052] Lustre: 966:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 779.548265] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 779.556086] Lustre: Skipped 2 previous similar messages [ 780.720533] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 781.607848] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 781.611350] Lustre: Skipped 1 previous similar message [ 782.275647] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 782.604384] Lustre: 23823:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734255713/real 1734255713] req@ffff88012fdbbb80 x1818498849352448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1734255729 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 783.102534] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 784.570989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 784.574290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 784.579215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 787.578356] Lustre: 23825:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734255718/real 1734255718] req@ffff88009b055500 x1818498849352704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1734255734 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 787.589095] Lustre: 23825:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 790.506096] Lustre: server umount lustre-MDT0000 complete [ 791.627852] LustreError: 976:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 791.634416] LustreError: 976:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 791.694274] LustreError: 25077:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255738 with bad export cookie 3153050845363207324 [ 791.695283] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 791.700431] LustreError: 25077:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 791.814081] Lustre: server umount lustre-MDT0001 complete [ 793.986977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 795.092357] Lustre: DEBUG MARKER: files compared the same [ 805.851477] Lustre: server umount lustre-OST0000 complete [ 812.445798] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 813.553041] Key type lgssc unregistered [ 813.714836] LNet: 3822:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.724988] LNet: Removed LNI 192.168.204.148@tcp [ 824.342056] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 824.649840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 824.670464] alg: No test for adler32 (adler32-zlib) [ 825.537605] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 825.625885] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 825.628732] LNet: Accept secure, port 988 [ 827.184372] Key type lgssc registered [ 827.558608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 830.926432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 830.963167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 832.406527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 833.856767] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug +malloc all [ 838.134363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 838.188758] Lustre: 6212:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 838.324981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 839.719903] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug +malloc all [ 841.987874] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 843.505360] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 846.590145] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 846.596489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 846.702335] Lustre: 7326:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 846.735690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 848.611374] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug +malloc all [ 850.669493] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 856.748392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000bd0:35 to 0x280000bd0:65) [ 866.762950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 866.762952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 866.763093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 866.763382] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 869.449358] Lustre: server umount lustre-OST0000 complete [ 871.771168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 871.772874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 871.772875] Lustre: Skipped 1 previous similar message [ 871.786270] Lustre: Skipped 1 previous similar message [ 876.779944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 876.784656] Lustre: Skipped 2 previous similar messages [ 877.171570] Lustre: server umount lustre-MDT0000 complete [ 878.592606] LustreError: 8129:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255825 with bad export cookie 2698191424627562699 [ 878.593417] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 878.601679] LustreError: 8129:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 878.744660] Lustre: server umount lustre-MDT0001 complete [ 882.696083] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 883.592923] Key type lgssc unregistered [ 883.684833] LNet: 9209:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.693363] LNet: Removed LNI 192.168.204.148@tcp [ 893.771900] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 893.994286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 894.042174] alg: No test for adler32 (adler32-zlib) [ 894.899653] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 894.989187] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 894.994978] LNet: Accept secure, port 988 [ 896.564216] Key type lgssc registered [ 897.081653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 900.172715] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 900.178554] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 900.185689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 917.328750] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 923.336549] LustreError: 10705:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 4: rc = -110 [ 953.360391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 955.147032] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 957.010072] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 959.682944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 959.759166] Lustre: *** cfs_fail_loc=706, val=0*** [ 959.928533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 959.953102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000bd0:35 to 0x280000bd0:97) [ 961.124499] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 964.423824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 964.477494] Lustre: 12819:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 964.620121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 966.048924] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 968.026852] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 969.393524] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 971.628070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 971.631060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 971.641233] Lustre: Skipped 1 previous similar message [ 972.518179] Lustre: server umount lustre-OST0000 complete [ 974.603867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 974.614227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 974.623209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 974.624945] Lustre: Skipped 1 previous similar message [ 976.635648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 976.636500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 980.690628] Lustre: server umount lustre-MDT0000 complete [ 981.645985] LustreError: 11957:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 981.653294] LustreError: 11957:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 982.433812] LustreError: 10713:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255928 with bad export cookie 13560208991459045460 [ 982.438800] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 982.445434] LustreError: 10713:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 982.607723] Lustre: server umount lustre-MDT0001 complete [ 986.079056] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 987.152919] Key type lgssc unregistered [ 987.289836] LNet: 14683:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.298096] LNet: Removed LNI 192.168.204.148@tcp [ 998.144496] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 998.314549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 998.325357] alg: No test for adler32 (adler32-zlib) [ 999.196241] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 999.281520] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 999.284987] LNet: Accept secure, port 988 [ 1000.847381] Key type lgssc registered [ 1001.220626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1004.041125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1004.050062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1006.022655] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1008.389787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1008.478661] Lustre: 17024:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1008.564741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1009.527171] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1012.325005] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1012.330922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1012.412678] Lustre: 17893:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1012.458836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1013.993762] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1016.410262] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1016.415546] Lustre: MGS: Client 132f8d75-8f7f-4e2e-b02e-887524016daa (at 0@lo) reconnecting [ 1016.489441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1016.507211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000bd0:35 to 0x280000bd0:129) [ 1017.375225] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1026.496349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1026.499522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1026.505019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1028.507031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1028.507435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1028.507600] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1031.040420] Lustre: server umount lustre-OST0000 complete [ 1033.516323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1033.516380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1033.517826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1033.517827] Lustre: Skipped 1 previous similar message [ 1038.447791] Lustre: server umount lustre-MDT0000 complete [ 1038.525625] LustreError: 17034:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1038.530881] LustreError: 17034:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 6 previous similar messages [ 1040.030065] Lustre: server umount lustre-MDT0001 complete [ 1041.446854] Lustre: server umount MGS complete [ 1045.747010] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 04:46:32 (1734255992) [ 1051.971668] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1052.718566] Key type lgssc unregistered [ 1052.806727] LNet: 21830:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1053.815562] LNet: Removed LNI 192.168.204.148@tcp [ 1058.802032] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1058.954066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1058.960806] alg: No test for adler32 (adler32-zlib) [ 1059.779857] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 1059.838611] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1059.840072] LNet: Accept secure, port 988 [ 1061.376626] Key type lgssc registered [ 1061.731114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1064.289760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1064.298639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1065.550738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1066.598919] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1070.118930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1070.182439] Lustre: 23947:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1070.299305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1071.369209] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1073.437072] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1074.757802] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1077.114516] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1077.124069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1077.220399] Lustre: 25057:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1077.282718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1079.132848] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1080.890763] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1091.284162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000bd0:131 to 0x280000bd0:161) [ 1100.331878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1100.336093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1100.341535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1101.290987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1101.292665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1101.299803] Lustre: Skipped 1 previous similar message [ 1104.429913] Lustre: server umount lustre-OST0000 complete [ 1106.299241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1106.299573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1106.305209] Lustre: Skipped 1 previous similar message [ 1111.307835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1111.313160] Lustre: Skipped 2 previous similar messages [ 1111.682838] Lustre: server umount lustre-MDT0000 complete [ 1112.925311] LustreError: 23069:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256059 with bad export cookie 16901759952604533383 [ 1112.926160] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1112.934346] LustreError: 23069:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1113.071589] Lustre: server umount lustre-MDT0001 complete [ 1116.743053] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1117.549096] Key type lgssc unregistered [ 1117.666782] LNet: 26940:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1118.673928] LNet: Removed LNI 192.168.204.148@tcp [ 1123.676436] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1123.817899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1123.844414] alg: No test for adler32 (adler32-zlib) [ 1124.666932] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 1124.724760] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1124.726682] LNet: Accept secure, port 988 [ 1126.262522] Key type lgssc registered [ 1126.551585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1128.749060] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1128.749835] LustreError: 28015:0:(tgt_mount.c:2178:server_fill_super()) cfs_race id 716 sleeping [ 1128.756232] Lustre: Skipped 1 previous similar message [ 1128.758420] LustreError: 28016:0:(tgt_mount.c:2178:server_fill_super()) cfs_fail_race id 716 waking [ 1128.763553] LustreError: 28015:0:(tgt_mount.c:2178:server_fill_super()) cfs_fail_race id 716 awake: rc=4986 [ 1128.766230] LustreError: 28015:0:(tgt_mount.c:2190:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1128.771481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1128.780072] LustreError: 28015:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1129.998577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1132.746737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1132.803944] Lustre: 28613:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1132.878334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1134.080338] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1135.592395] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing lsmod [ 1136.545023] LustreError: 29420:0:(tgt_mount.c:2178:server_fill_super()) cfs_race id 716 sleeping [ 1136.549946] LustreError: 29421:0:(tgt_mount.c:2178:server_fill_super()) cfs_fail_race id 716 waking [ 1136.553265] LustreError: 29420:0:(tgt_mount.c:2178:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 1136.556451] LustreError: 29420:0:(tgt_mount.c:2190:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1136.557296] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1136.561080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1136.571906] LustreError: 29420:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1136.653411] Lustre: 29421:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1136.700426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1137.793039] LustreError: 28087:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1137.798777] LustreError: 28087:0:(lod_sub_object.c:914:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1137.803072] LustreError: 28087:0:(lod_dev.c:506:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 1137.868122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1137.872428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.879015] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1141.707674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1141.707917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.716156] Lustre: Skipped 2 previous similar messages [ 1142.715363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1142.721023] Lustre: Skipped 2 previous similar messages [ 1143.874185] Lustre: server umount lustre-MDT0000 complete [ 1145.206763] LustreError: 29663:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256091 with bad export cookie 8884440239139863412 [ 1145.208117] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1145.221367] LustreError: 29663:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1146.768421] LustreError: 28622:0:(client.c:1300:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88009f4a1500 x1818499293920384/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 1146.854046] Lustre: server umount lustre-MDT0001 complete [ 1158.267843] Lustre: server umount lustre-OST0000 complete [ 1162.357183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1162.612056] LustreError: 30832:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1162.620773] LustreError: 30832:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 1162.661171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1163.803034] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1167.148849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1167.212690] Lustre: 31692:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1167.334608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1168.784094] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1170.530659] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1171.555639] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1173.741264] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1173.746263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1173.823942] Lustre: 335:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1175.614967] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1177.313455] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1187.876962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000bd0:131 to 0x280000bd0:193) [ 1196.906920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1196.908841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.912603] Lustre: Skipped 1 previous similar message [ 1196.914504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1199.349067] Lustre: server umount lustre-OST0000 complete [ 1202.362950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1202.366048] LustreError: Skipped 1 previous similar message [ 1202.367381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1202.371830] Lustre: Skipped 1 previous similar message [ 1202.373685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1202.376187] Lustre: Skipped 2 previous similar messages [ 1206.547845] Lustre: server umount lustre-MDT0000 complete [ 1207.630140] LustreError: 30814:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256154 with bad export cookie 8884440239139864091 [ 1207.631177] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1207.639302] LustreError: 30814:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1207.752421] Lustre: server umount lustre-MDT0001 complete [ 1211.277282] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1211.909554] Key type lgssc unregistered [ 1211.996548] LNet: 2262:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1213.001747] LNet: Removed LNI 192.168.204.148@tcp [ 1220.407706] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1220.526583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1220.533885] alg: No test for adler32 (adler32-zlib) [ 1221.368866] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 1221.432081] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1221.433931] LNet: Accept secure, port 988 [ 1222.973497] Key type lgssc registered [ 1223.216798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1225.689376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1225.697877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1226.881016] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1227.856849] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1230.772226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1230.836670] Lustre: 4650:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1230.947881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1232.007119] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1234.025719] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1235.216235] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1237.534642] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1237.539886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1237.623828] Lustre: 5760:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1237.668306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1239.354921] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1240.946329] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1251.674466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000bd0:195 to 0x280000bd0:225) [ 1257.766164] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 1258.383526] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 1260.699463] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1260.703475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1260.708661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1261.690855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1261.690934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1261.697769] Lustre: Skipped 1 previous similar message [ 1265.200841] Lustre: server umount lustre-OST0000 complete [ 1266.698971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1266.703525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1270.987263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1270.990482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1270.995708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1272.275041] Lustre: server umount lustre-MDT0000 complete [ 1273.373175] LustreError: 6504:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256219 with bad export cookie 7498885011515866860 [ 1273.374280] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1273.380358] LustreError: 6504:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1273.481412] Lustre: server umount lustre-MDT0001 complete [ 1276.543047] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1277.136417] Key type lgssc unregistered [ 1277.228830] LNet: 7783:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1278.234699] LNet: Removed LNI 192.168.204.148@tcp [ 1283.396185] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1283.523276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1283.536989] alg: No test for adler32 (adler32-zlib) [ 1284.345094] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 1284.396200] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1284.398269] LNet: Accept secure, port 988 [ 1285.931378] Key type lgssc registered [ 1286.170384] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1288.208274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1288.216272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1289.392443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1290.346971] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1292.998858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1293.046395] Lustre: 9901:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1293.134739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1294.035374] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1295.500225] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1296.576266] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1298.732736] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1298.738573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1298.828038] Lustre: 11014:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1298.883687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1299.885947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000bd0:227 to 0x280000bd0:257) [ 1300.513583] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1302.128440] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1305.663290] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1305.667179] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1305.675599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1307.899921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1307.902073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1307.902085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1307.911921] Lustre: Skipped 1 previous similar message [ 1311.626804] Lustre: server umount lustre-OST0000 complete [ 1313.147407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1313.150572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.156676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1317.915178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1317.921650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1319.118061] Lustre: server umount lustre-MDT0000 complete [ 1320.733562] LustreError: 9023:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256267 with bad export cookie 7601565023292772130 [ 1320.735494] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1320.743360] LustreError: 9023:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1320.890268] Lustre: server umount lustre-MDT0001 complete [ 1325.072342] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1325.861134] Key type lgssc unregistered [ 1325.955183] LNet: 12891:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.962773] LNet: Removed LNI 192.168.204.148@tcp [ 1334.751809] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1334.926293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1334.941424] alg: No test for adler32 (adler32-zlib) [ 1335.774653] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 1335.847619] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1335.851295] LNet: Accept secure, port 988 [ 1337.395370] Key type lgssc registered [ 1337.660832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1340.100844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1340.109151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1341.299904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1342.218278] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1345.067015] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1345.119021] Lustre: 15266:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1345.212293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1346.112078] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1347.707932] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1348.918362] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1350.939346] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1350.945708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1351.028425] Lustre: 16378:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1351.075744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1352.077134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000bd0:259 to 0x280000bd0:289) [ 1352.552971] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1353.971427] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1357.221667] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 1358.305947] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 1358.308371] Lustre: Skipped 2 previous similar messages [ 1359.462839] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 1359.465826] Lustre: Skipped 2 previous similar messages [ 1360.503015] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1360.509422] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1362.893091] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 1362.896225] Lustre: Skipped 2 previous similar messages [ 1366.614033] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.48@tcp 192.168.0.[2,10]@tcp [ 1369.913613] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 1369.918108] Lustre: Skipped 2 previous similar messages [ 1375.858006] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.48/24@tcp [ 1375.861462] Lustre: Skipped 1 previous similar message [ 1380.123317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1380.123455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1380.123724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1380.131548] Lustre: Skipped 1 previous similar message [ 1384.427366] Lustre: server umount lustre-OST0000 complete [ 1390.139523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1390.139975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1390.139976] Lustre: Skipped 1 previous similar message [ 1390.149277] Lustre: Skipped 1 previous similar message [ 1391.536183] Lustre: server umount lustre-MDT0000 complete [ 1392.771430] LustreError: 17241:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256339 with bad export cookie 3237443745810788050 [ 1392.772982] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1392.779536] LustreError: 17241:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1392.897343] Lustre: server umount lustre-MDT0001 complete [ 1396.267058] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1397.080780] Key type lgssc unregistered [ 1397.184984] LNet: 20177:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1398.191807] LNet: Removed LNI 192.168.204.148@tcp [ 1406.436622] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1406.580632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1406.621320] alg: No test for adler32 (adler32-zlib) [ 1407.453149] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 1407.507094] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1407.509515] LNet: Accept secure, port 988 [ 1409.060397] Key type lgssc registered [ 1409.344723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1411.934199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1411.945450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1413.168059] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1413.179075] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.48/24@tcp [ 1413.192922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1414.282706] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1417.524509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1417.578588] Lustre: 22749:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1417.652630] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1417.660481] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.48/24@tcp [ 1417.677193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1418.664915] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1420.428259] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1421.825157] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1424.661392] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1424.666546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1424.755987] Lustre: 23861:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1424.825609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1426.712819] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1428.682039] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1437.824591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000bd0:297 to 0x280000bd0:321) [ 1442.828452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1442.828457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1442.837322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1442.843889] Lustre: Skipped 1 previous similar message [ 1447.590912] Lustre: server umount lustre-OST0000 complete [ 1452.700558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1452.704169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1452.709783] Lustre: Skipped 1 previous similar message [ 1452.711869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1454.937358] Lustre: server umount lustre-MDT0000 complete [ 1456.291165] LustreError: 21871:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256402 with bad export cookie 14928537018732983029 [ 1456.294014] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1456.299262] LustreError: 21871:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1456.416170] Lustre: server umount lustre-MDT0001 complete [ 1460.273699] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1461.176433] Key type lgssc unregistered [ 1461.285740] LNet: 25809:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1462.295262] LNet: Removed LNI 192.168.204.148@tcp [ 1471.540316] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1471.763486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1471.804199] alg: No test for adler32 (adler32-zlib) [ 1472.647669] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 1472.712730] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1472.714957] LNet: Accept secure, port 988 [ 1474.254547] Key type lgssc registered [ 1474.599644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1477.239162] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1477.248730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1478.442129] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1478.450905] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.48/24@tcp [ 1478.453712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1479.504227] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1482.427579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1482.477910] Lustre: 28253:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1482.549264] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1482.556615] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.48/24@tcp [ 1482.571727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1483.566828] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1485.067098] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1486.106105] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1488.463941] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1488.470263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1488.556029] Lustre: 29364:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1488.621016] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1490.138356] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1491.763691] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1502.624271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000bd0:323 to 0x280000bd0:353) [ 1522.651564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1522.651566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1522.651970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1522.660897] Lustre: Skipped 1 previous similar message [ 1524.170122] Lustre: server umount lustre-OST0000 complete [ 1527.611563] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1527.617152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1527.625057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1527.627885] Lustre: Skipped 1 previous similar message [ 1531.396348] Lustre: server umount lustre-MDT0000 complete [ 1532.667482] LustreError: 27392:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1532.674215] LustreError: 27392:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 3 previous similar messages [ 1532.718969] LustreError: 27373:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256479 with bad export cookie 2067484897974487065 [ 1532.721885] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1532.738205] LustreError: 27373:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1532.869187] Lustre: server umount lustre-MDT0001 complete [ 1536.646036] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1537.455575] Key type lgssc unregistered [ 1537.552157] LNet: 31255:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1538.561815] LNet: Removed LNI 192.168.204.148@tcp [ 1545.702173] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1545.834447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1545.846063] alg: No test for adler32 (adler32-zlib) [ 1546.667384] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 1546.722953] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1546.724746] LNet: Accept secure, port 988 [ 1548.263369] Key type lgssc registered [ 1548.504432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1550.571967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1550.578085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1551.709439] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1551.714337] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.48/24@tcp [ 1551.716932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1552.524203] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1554.863993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1554.904649] Lustre: 1207:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1554.968595] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1554.974210] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.48/24@tcp [ 1554.987220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1555.803474] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1557.242046] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1558.109406] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1559.993921] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1560.001099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1560.088781] Lustre: 2320:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1560.133029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1561.480042] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1562.132901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000bd0:355 to 0x280000bd0:385) [ 1562.823658] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1572.155151] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1572.159184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1572.164010] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1573.958724] Lustre: lustre-OST0000: Not available for connect from 192.168.204.48@tcp (stopping) [ 1575.147078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1575.147298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1575.153104] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1577.026217] Lustre: server umount lustre-OST0000 complete [ 1578.794938] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8800b5813800.ost_server_uuid 50 [ 1579.366224] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b5813800.ost_server_uuid in DISCONN state after 0 sec [ 1579.936064] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 1589.863506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1589.865946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1589.865947] Lustre: Skipped 1 previous similar message [ 1589.876379] Lustre: Skipped 1 previous similar message [ 1594.874854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1594.877242] Lustre: Skipped 2 previous similar messages [ 1595.927891] Lustre: server umount lustre-MDT0000 complete [ 1597.029322] LustreError: 3661:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256543 with bad export cookie 1870390231964514136 [ 1597.031019] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1597.037759] LustreError: 3661:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1597.138872] Lustre: server umount lustre-MDT0001 complete [ 1599.214433] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 04:55:45 (1734256545) [ 1603.525596] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 1605.386321] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1608.266664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1610.300898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1612.472331] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1612.480588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1614.654188] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1614.664775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1617.090693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1617.115619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1617.225676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1617.238704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1617.279520] Lustre: lustre-MDT0000: new disk, initializing [ 1617.317628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1617.327505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1618.417376] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1622.838611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1622.869210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1622.919707] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1622.938030] Lustre: 8607:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1622.957886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1622.960499] Lustre: Skipped 1 previous similar message [ 1623.001873] Lustre: lustre-MDT0001: new disk, initializing [ 1623.044149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1623.065062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1623.069811] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1624.362984] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1627.643797] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1628.633767] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1631.046637] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1631.054168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1631.085220] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1631.098015] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1631.211022] Lustre: 9802:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1631.225849] Lustre: lustre-OST0000: new disk, initializing [ 1631.228472] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1631.258094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1632.075545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1632.080193] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1632.108751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1633.057920] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1636.252884] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1637.425140] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1637.493959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1638.582999] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1638.674606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1642.107186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1642.107228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1642.107782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1642.107783] Lustre: Skipped 2 previous similar messages [ 1642.119599] LustreError: Skipped 1 previous similar message [ 1645.347792] Lustre: server umount lustre-OST0000 complete [ 1647.115073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1647.120812] Lustre: Skipped 2 previous similar messages [ 1652.646400] Lustre: server umount lustre-MDT0000 complete [ 1653.753123] LustreError: 11478:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256600 with bad export cookie 1870390231964515193 [ 1653.754685] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1653.761785] LustreError: 11478:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1653.867199] Lustre: server umount lustre-MDT0001 complete [ 1656.374731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1656.536628] LustreError: 12368:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1656.546206] LustreError: 12368:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 1656.564527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1657.504782] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1660.331572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1660.381904] Lustre: 13231:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1661.613418] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1663.277714] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1664.726529] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1667.206577] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1667.215752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1669.000635] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1670.577385] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1671.544917] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1682.853912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 11 sec [ 1684.044412] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1684.116194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1685.387820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1685.391951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1685.399572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1685.404261] Lustre: Skipped 5 previous similar messages [ 1690.832544] Lustre: server umount lustre-OST0000 complete [ 1692.551211] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1693.793273] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1716.959224] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1716.965648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1717.047587] Lustre: 16855:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1717.051990] Lustre: 16855:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [ 1717.102497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1717.109463] Lustre: Skipped 2 previous similar messages [ 1718.774999] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1720.583607] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1721.155356] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 1727.086146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1727.090643] Lustre: Skipped 2 previous similar messages [ 1728.147524] Lustre: server umount lustre-OST0000 complete [ 1730.573705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1730.577460] LustreError: Skipped 1 previous similar message [ 1730.580565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1730.588584] Lustre: Skipped 1 previous similar message [ 1735.465217] Lustre: server umount lustre-MDT0000 complete [ 1736.866029] LustreError: 12349:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256683 with bad export cookie 1870390231964516061 [ 1736.867818] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1736.879861] LustreError: 12349:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1737.002807] Lustre: server umount lustre-MDT0001 complete [ 1739.669484] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 04:58:06 (1734256686) [ 1739.987152] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1739.992315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1742.139503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1742.362801] LustreError: 19197:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1743.466581] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1746.058004] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1746.093602] Lustre: 20058:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1747.075652] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1748.680055] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1749.810631] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1752.071216] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1752.076702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1752.096801] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1752.102739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1752.188384] Lustre: Found index 0 for lustre-OST0000, updating log [ 1752.200353] LustreError: 21179:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1752.213592] Lustre: Permanently deactivating lustre-OST0000 [ 1752.221854] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [ 1753.902243] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1754.989317] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1755.015349] LustreError: 19215:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 1755.024070] LustreError: 19215:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 1755.842117] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1758.426439] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1758.433446] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1758.479220] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1758.486811] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1758.549884] Lustre: lustre-OST0001: new disk, initializing [ 1758.553801] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1760.169050] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1761.372809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1761.376508] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1761.399514] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1763.031495] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1763.520219] Lustre: Permanently reactivating lustre-OST0000 [ 1763.525900] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 1763.528218] Lustre: Skipped 2 previous similar messages [ 1768.023942] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1768.029391] Lustre: Skipped 1 previous similar message [ 1768.032289] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1768.037729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1768.043902] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 1771.404270] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1777.191038] Lustre: server umount lustre-OST0001 complete [ 1781.421936] LustreError: 21179:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1781.429923] LustreError: 21179:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 1784.791956] Lustre: server umount lustre-OST0000 complete [ 1791.227556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1791.231986] LustreError: Skipped 1 previous similar message [ 1791.235056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1791.237401] Lustre: Skipped 10 previous similar messages [ 1792.452689] Lustre: server umount lustre-MDT0000 complete [ 1793.669463] LustreError: 20073:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256740 with bad export cookie 1870390231964516992 [ 1793.672276] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1793.677470] LustreError: 20073:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1793.855188] Lustre: server umount lustre-MDT0001 complete [ 1797.713596] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1798.661518] Key type lgssc unregistered [ 1798.788367] LNet: 24607:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.801718] LNet: Removed LNI 192.168.204.148@tcp [ 1807.720527] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1807.855487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1807.860083] alg: No test for adler32 (adler32-zlib) [ 1808.710428] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 1808.792078] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1808.794227] LNet: Accept secure, port 988 [ 1810.347308] Key type lgssc registered [ 1810.721913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1811.918536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1814.277996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1814.283737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1815.526266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1816.717999] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1820.094010] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1820.118870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1820.170318] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1820.193648] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1820.197074] Lustre: Permanently deactivating lustre-MDT0001 [ 1820.227544] Lustre: 27055:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1820.378303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1821.744369] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1823.524647] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 1823.529090] LustreError: 26222:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1823.541373] LustreError: 26222:0:(lod_sub_object.c:914:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1823.556199] LustreError: 26222:0:(lod_dev.c:506:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 1, failed: rc = -108 [ 1824.230422] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1825.790966] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1828.664812] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1828.670230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1828.765609] Lustre: 28187:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1828.836729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1829.834262] LustreError: 28194:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1829.847136] LustreError: 28194:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 1829.856348] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1830.862083] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1832.992330] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1834.843758] LustreError: 28384:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1835.601741] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1835.612243] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1835.660426] Lustre: 29304:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1835.722489] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1837.550405] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1839.496504] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1840.723701] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 1846.094235] Lustre: Permanently reactivating lustre-MDT0001 [ 1846.101819] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 1846.105037] Lustre: Skipped 3 previous similar messages [ 1849.444779] Lustre: lustre-MDT0001: Client d5063ccc-56f2-4b84-a8e2-ac9c45bbb87a (at 192.168.204.48@tcp) reconnecting [ 1851.272360] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1851.791002] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1853.250190] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 1853.324766] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 1854.446825] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 1854.518002] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 1855.467142] Lustre: Permanently deactivating lustre-MDT0001 [ 1855.475715] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 1855.478086] Lustre: Skipped 2 previous similar messages [ 1858.250808] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 1865.404716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1865.410827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1865.417459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1870.574689] Lustre: server umount lustre-MDT0000 complete [ 1870.763404] LustreError: 27686:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1870.772424] LustreError: 27686:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 3 previous similar messages [ 1871.948359] LustreError: 30962:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256818 with bad export cookie 16816167152903623007 [ 1871.950316] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1871.956665] LustreError: 30962:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1872.085590] Lustre: server umount lustre-MDT0001 complete [ 1879.626576] Lustre: server umount lustre-OST0000 complete [ 1883.873147] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 05:00:30 (1734256830) [ 1886.536362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1896.775675] LustreError: 25296:0:(client.c:1310:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88012db9ca80 x1818500011190784/t0(0) o250->MGC192.168.204.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 1896.901180] LustreError: 28203:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1896.919779] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 1896.944481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1896.956007] LustreError: 428:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -108 [ 1896.960487] LustreError: 428:0:(osp_object.c:617:osp_attr_get()) Skipped 1 previous similar message [ 1896.965499] LustreError: 428:0:(lod_sub_object.c:914:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 1896.971067] LustreError: 428:0:(lod_sub_object.c:914:lod_sub_prep_llog()) Skipped 1 previous similar message [ 1896.979227] LustreError: 428:0:(lod_dev.c:506:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -108 [ 1896.985584] LustreError: 428:0:(lod_dev.c:346:lod_sub_recreate_llog()) lustre-MDT0000-mdtlov: can't access update_log: rc = -108 [ 1898.082502] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1900.938938] LustreError: 28192:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1900.948602] LustreError: 28192:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 1901.152066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1901.207303] Lustre: 1298:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1901.362621] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:65) [ 1902.533026] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1904.525697] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1905.796780] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1908.297778] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1908.304380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1908.354702] Lustre: 2416:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1908.446326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1908.448920] Lustre: Skipped 1 previous similar message [ 1909.492272] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 1910.250557] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1912.105676] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1920.498802] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.204.148@tcp (at 0@lo) [ 1949.563630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1949.563631] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1949.563642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.563643] Lustre: Skipped 3 previous similar messages [ 1949.564723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1949.564724] Lustre: Skipped 3 previous similar messages [ 1955.384814] Lustre: server umount lustre-OST0000 complete [ 1955.547554] LustreError: 28192:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1955.559486] LustreError: 28192:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 5 previous similar messages [ 1957.328894] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1957.335343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 1962.543652] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1962.552324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1962.603552] Lustre: 4482:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1962.685210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1964.816112] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1967.217648] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1967.694905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1967.703082] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1967.705258] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 2002.751567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2002.760741] Lustre: Skipped 1 previous similar message [ 2002.764992] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2002.765411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2002.765413] Lustre: Skipped 3 previous similar messages [ 2007.014100] Lustre: server umount lustre-OST0000 complete [ 2007.757199] LustreError: 28192:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2007.766977] LustreError: 28192:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 2011.469235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2011.472821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.491707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2011.501285] Lustre: Skipped 1 previous similar message [ 2014.748096] Lustre: server umount lustre-MDT0000 complete [ 2016.488367] LustreError: 30962:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256963 with bad export cookie 16816167152903625345 [ 2016.495296] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2016.514431] LustreError: 30962:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2022.669753] Lustre: server umount lustre-MDT0001 complete [ 2028.083701] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2031.565547] Lustre: DEBUG MARKER: == conf-sanity test 53a: check OSS thread count params === 05:02:57 (1734256977) [ 2038.598792] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2042.059160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2061.784786] LustreError: 25296:0:(client.c:1310:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8800a6209880 x1818500011256576/t0(0) o250->MGC192.168.204.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 2061.928164] LustreError: 28202:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2061.934305] LustreError: 28202:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 3 previous similar messages [ 2061.949933] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 2061.983474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2061.993560] LustreError: 7530:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -108 [ 2061.999231] LustreError: 7530:0:(osp_object.c:617:osp_attr_get()) Skipped 1 previous similar message [ 2062.003222] LustreError: 7530:0:(lod_sub_object.c:914:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 2062.010658] LustreError: 7530:0:(lod_dev.c:506:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -108 [ 2062.018675] LustreError: 7530:0:(lod_dev.c:346:lod_sub_recreate_llog()) lustre-MDT0000-mdtlov: can't access update_log: rc = -108 [ 2062.031903] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:74 to 0x2c0000401:97) [ 2063.270145] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2066.674838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2066.737223] Lustre: 8367:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2066.870987] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:97) [ 2067.874713] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2069.999005] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2071.230833] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2073.989320] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2073.998232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2076.126295] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2077.070324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:138 to 0x280000401:161) [ 2077.071227] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:161) [ 2078.030764] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2078.072935] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.204.148@tcp (at 0@lo) [ 2078.077606] Lustre: Skipped 1 previous similar message [ 2085.977619] Lustre: DEBUG MARKER: conf-sanity test_53a: @@@@@@ FAIL: client_up failed [ 2089.453343] Lustre: DEBUG MARKER: == conf-sanity test 53b: check MDS thread count params === 05:03:56 (1734257036) [ 2092.356573] Lustre: DEBUG MARKER: conf-sanity test_53b: @@@@@@ FAIL: MDT start failed [ 2094.230573] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 05:04:00 (1734257040) [ 2097.400197] Lustre: DEBUG MARKER: conf-sanity test_54b: @@@@@@ FAIL: MDT start failed [ 2099.334546] Lustre: DEBUG MARKER: == conf-sanity test 58: missing llog files must not prevent MDT from mounting ========================================================== 05:04:05 (1734257045) [ 2111.898980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2111.902080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2111.907048] Lustre: Skipped 2 previous similar messages [ 2111.909465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2111.912140] Lustre: Skipped 3 previous similar messages [ 2115.154429] Lustre: server umount lustre-MDT0000 complete [ 2116.501278] LustreError: 30962:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257063 with bad export cookie 16816167152903628656 [ 2116.504157] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2116.513650] LustreError: 30962:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2122.693627] Lustre: server umount lustre-MDT0001 complete [ 2124.929515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2127.908716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2142.166750] LustreError: 25296:0:(client.c:1310:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff880093823480 x1818500011294336/t0(0) o250->MGC192.168.204.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 2142.271142] LustreError: 28203:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2142.280587] LustreError: 28203:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 11 previous similar messages [ 2142.285428] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 2142.313846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2142.318558] Lustre: Skipped 2 previous similar messages [ 2142.330824] LustreError: 14599:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -108 [ 2142.334847] LustreError: 14599:0:(osp_object.c:617:osp_attr_get()) Skipped 1 previous similar message [ 2142.339200] LustreError: 14599:0:(lod_sub_object.c:914:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 2142.341593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:212 to 0x280000401:257) [ 2142.348995] LustreError: 14599:0:(lod_dev.c:506:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -108 [ 2142.352537] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:148 to 0x2c0000401:193) [ 2142.360891] LustreError: 14599:0:(lod_dev.c:346:lod_sub_recreate_llog()) lustre-MDT0000-mdtlov: can't access update_log: rc = -108 [ 2143.189772] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2145.604828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2145.642365] Lustre: 15433:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2145.646337] Lustre: 15433:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [ 2145.739134] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:193) [ 2145.739794] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:129) [ 2146.590166] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2148.113933] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2148.550153] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2149.430350] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2150.729859] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) [ 3660.717938] Lustre: DEBUG MARKER: rpc test_58: @@@@@@ FAIL: can't put import for mdc.lustre-MDT0001-mdc-*.mds_server_uuid into FULL state after 1475 sec, have DISCONN [ 3663.352098] Lustre: DEBUG MARKER: conf-sanity test_58: @@@@@@ FAIL: import is not in FULL state [ 3664.759225] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 05:30:11 (1734258611) [ 3666.872705] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 05:30:13 (1734258613) [ 3670.057916] Lustre: DEBUG MARKER: conf-sanity test_61a: @@@@@@ FAIL: MDT start failed [ 3671.601098] Lustre: DEBUG MARKER: == conf-sanity test 61b: large xattr ===================== 05:30:18 (1734258618) [ 3674.736668] Lustre: DEBUG MARKER: conf-sanity test_61b: @@@@@@ FAIL: MDT start failed [ 3676.132302] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 3676.657877] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 05:30:23 (1734258623) [ 3678.202900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3678.207153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3678.212961] Lustre: Skipped 3 previous similar messages [ 3678.215678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3678.218266] Lustre: Skipped 5 previous similar messages [ 3680.790578] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.48@tcp (stopping) [ 3682.203063] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3682.208326] Lustre: Skipped 1 previous similar message [ 3683.669890] Lustre: server umount lustre-MDT0000 complete [ 3684.774192] LustreError: 30962:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734258631 with bad export cookie 16816167152903636692 [ 3684.776089] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3684.782854] LustreError: 30962:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3684.892898] Lustre: server umount lustre-MDT0001 complete [ 3686.136497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3688.510262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3695.211827] LustreError: 25296:0:(client.c:1310:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8800987e0000 x1818500011865984/t0(0) o250->MGC192.168.204.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 3695.318713] Lustre: lustre-MDT0000: new disk, initializing [ 3695.336322] LustreError: 28203:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3695.356080] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3695.377065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3695.380598] Lustre: Skipped 1 previous similar message [ 3695.389963] LustreError: 21619:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -108 [ 3695.395636] LustreError: 21619:0:(osp_object.c:617:osp_attr_get()) Skipped 1 previous similar message [ 3695.400016] LustreError: 21619:0:(lod_sub_object.c:914:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 3695.401999] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:212 to 0x280000401:289) [ 3695.402044] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:148 to 0x2c0000401:225) [ 3695.413607] LustreError: 21619:0:(lod_dev.c:506:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -108 [ 3695.422193] LustreError: 21619:0:(lod_dev.c:346:lod_sub_recreate_llog()) lustre-MDT0000-mdtlov: can't access update_log: rc = -108 [ 3696.355346] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3698.961727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3699.008058] Lustre: 22453:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.148@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3699.124860] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:225) [ 3699.124975] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:161) [ 3700.103139] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3701.675603] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3702.111925] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) [ 3702.114410] Lustre: Skipped 1 previous similar message [ 3702.124570] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3703.028570] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5201.234784] Lustre: DEBUG MARKER: rpc test_65: @@@@@@ FAIL: can't put import for mdc.lustre-MDT0001-mdc-*.mds_server_uuid into FULL state after 1475 sec, have DISCONN