-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:37:09 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg333-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg333-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg333-server oleg333-server: oleg333-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing load_modules_local oleg333-server: Loading modules from /home/green/git/lustre-release/lustre oleg333-server: detected 4 online CPUs by sysfs oleg333-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg333-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg333-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg333-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg333-server: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg333-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg333-server: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg333-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg333-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg333-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:38:37 (1709786317) start mds service on oleg333-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg333-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg333-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-OST0000 oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg333-client.virtnet: -o user_xattr,flock oleg333-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h38m57s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800b6331800.stats=clear mdc.lustre-MDT0001-mdc-ffff8800b6331800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:39:42.000000000 -0500 Modify: 2024-03-06 23:39:42.000000000 -0500 Change: 2024-03-06 23:39:42.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 85 lines, 85 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg333-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg333-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg333-server unloading modules on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing unload_modules_local modules unloaded. oleg333-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg333-server: Is the ldiskfs module available? oleg333-server: oleg333-server: tunefs.lustre FATAL: failed to write local files oleg333-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg333-client: oleg333-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg333-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg333-server: Is the ldiskfs module available? oleg333-server: oleg333-server: tunefs.lustre FATAL: failed to write local files oleg333-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg333-client: oleg333-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.133@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.133@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg333-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg333-server: Is the ldiskfs module available? oleg333-server: oleg333-server: tunefs.lustre FATAL: failed to write local files oleg333-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg333-client: oleg333-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.133@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.133@tcp sys.timeout=20 oleg333-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg333-server: Is the ldiskfs module available? oleg333-server: oleg333-server: tunefs.lustre FATAL: failed to write local files oleg333-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg333-client: oleg333-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.133@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.133@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2 oleg333-server: oleg333-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing load_modules_local oleg333-server: Loading modules from /home/green/git/lustre-release/lustre oleg333-server: detected 4 online CPUs by sysfs oleg333-server: Force libcfs to create 2 CPU partitions oleg333-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg333-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg333-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg333-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg333-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg333-server: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg333-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg333-server: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg333-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg333-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg333-server PASS 35b (147s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:41:06 (1709786466) start mds service on oleg333-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg333-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg333-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-OST0000 oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg333-client.virtnet: -o user_xattr,flock oleg333-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg333-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg333-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg333-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg333-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg333-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg333-client.virtnet: -o user_xattr,flock oleg333-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg333-client.virtnet /mnt/lustre (opts:) stop mds service on oleg333-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg333-server 000000 2 000008 oleg333-server: 8+0 records in oleg333-server: 8+0 records out oleg333-server: 4096 bytes (4.1 kB) copied, 0.00101614 s, 4.0 MB/s start mds service on oleg333-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg333-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg333-client.virtnet: -o user_xattr,flock oleg333-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg333-client.virtnet /mnt/lustre (opts:) stop mds service on oleg333-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg333-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg333-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server stop mds service on oleg333-server stop mds service on oleg333-server unloading modules on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2 PASS 38 (120s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg333-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg333-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg333-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2 oleg333-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg333-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:43:09 (1709786589) Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2 oleg333-server: oleg333-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing load_modules_local oleg333-server: Loading modules from /home/green/git/lustre-release/lustre oleg333-server: detected 4 online CPUs by sysfs oleg333-server: Force libcfs to create 2 CPU partitions oleg333-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg333-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg333-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg333-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg333-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg333-client.virtnet: -o user_xattr,flock oleg333-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg333-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec stop ost2 service on oleg333-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg333-server umount lustre on /mnt/lustre..... Stopping client oleg333-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg333-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg333-server unloading modules on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing unload_modules_local modules unloaded. oleg333-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg333-server: Is the ldiskfs module available? oleg333-server: oleg333-server: tunefs.lustre FATAL: failed to write local files oleg333-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg333-client: oleg333-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg333-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg333-server: Is the ldiskfs module available? oleg333-server: oleg333-server: tunefs.lustre FATAL: failed to write local files oleg333-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg333-client: oleg333-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.133@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.203.133@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg333-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg333-server: Is the ldiskfs module available? oleg333-server: oleg333-server: tunefs.lustre FATAL: failed to write local files oleg333-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg333-client: oleg333-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.133@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.133@tcp sys.timeout=20 oleg333-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg333-server: Is the ldiskfs module available? oleg333-server: oleg333-server: tunefs.lustre FATAL: failed to write local files oleg333-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg333-client: oleg333-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.133@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.133@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2 oleg333-server: oleg333-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing load_modules_local oleg333-server: Loading modules from /home/green/git/lustre-release/lustre oleg333-server: detected 4 online CPUs by sysfs oleg333-server: Force libcfs to create 2 CPU partitions oleg333-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg333-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg333-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg333-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg333-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg333-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg333-server: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg333-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg333-server: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg333-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg333-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg333-server PASS 51 (211s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:46:43 (1709786803) start mds service on oleg333-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg333-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg333-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-OST0000 oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg333-client.virtnet: -o user_xattr,flock oleg333-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 327 (time 1709786845.62 total 10.00 last 32.69) - open/close 658 (time 1709786855.62 total 20.01 last 33.08) - open/close 870 (time 1709786866.26 total 30.65 last 19.92) - open/close 1216 (time 1709786877.25 total 41.64 last 31.49) - open/close 1625 (time 1709786887.90 total 52.28 last 38.41) - open/close 1849 (time 1709786898.59 total 62.98 last 20.94) - open/close 2111 (time 1709786908.60 total 72.98 last 26.20) - open/close 2478 (time 1709786919.40 total 83.79 last 33.96) - open/close 2708 (time 1709786930.22 total 94.61 last 21.26) - open/close 2989 (time 1709786940.23 total 104.61 last 28.08) - open/close 3174 (time 1709786950.23 total 114.62 last 18.49) - open/close 3315 (time 1709786960.25 total 124.63 last 14.08) - open/close 3432 (time 1709786970.25 total 134.63 last 11.70) - open/close 3587 (time 1709786982.46 total 146.85 last 12.69) - open/close 3728 (time 1709786994.65 total 159.03 last 11.57) - open/close 3945 (time 1709787004.78 total 169.16 last 21.43) - open/close 4177 (time 1709787014.78 total 179.16 last 23.20) - open/close 4314 (time 1709787025.01 total 189.40 last 13.38) - open/close 4528 (time 1709787035.29 total 199.67 last 20.83) - open/close 4777 (time 1709787046.23 total 210.62 last 22.75) - open/close 4996 (time 1709787057.01 total 221.40 last 20.31) - open/close 5058 (time 1709787070.86 total 235.25 last 4.48) - open/close 5297 (time 1709787081.71 total 246.10 last 22.02) - open/close 5497 (time 1709787091.73 total 256.12 last 19.96) - open/close 5668 (time 1709787101.99 total 266.37 last 16.68) - open/close 5867 (time 1709787112.93 total 277.32 last 18.18) - open/close 6118 (time 1709787122.94 total 287.32 last 25.08) - open/close 6259 (time 1709787133.90 total 298.28 last 12.87) - open/close 6446 (time 1709787144.72 total 309.10 last 17.28) - open/close 6623 (time 1709787155.63 total 320.01 last 16.23) - open/close 6889 (time 1709787166.10 total 330.49 last 25.39) - open/close 7080 (time 1709787176.72 total 341.10 last 17.99) - open/close 7244 (time 1709787186.73 total 351.11 last 16.38) - open/close 7348 (time 1709787197.03 total 361.41 last 10.10) - open/close 7460 (time 1709787207.24 total 371.63 last 10.96) - open/close 7690 (time 1709787218.00 total 382.39 last 21.37) - open/close 7748 (time 1709787232.70 total 397.09 last 3.95) - open/close 7933 (time 1709787243.04 total 407.43 last 17.89) - open/close 8083 (time 1709787253.18 total 417.57 last 14.80) - open/close 8242 (time 1709787263.53 total 427.91 last 15.37) - open/close 8512 (time 1709787273.54 total 437.92 last 26.97) - open/close 8704 (time 1709787283.54 total 447.92 last 19.20) - open/close 8846 (time 1709787293.54 total 457.93 last 14.19) - open/close 8974 (time 1709787304.26 total 468.64 last 11.95) - open/close 8998 (time 1709787315.86 total 480.25 last 2.07) - open/close 9128 (time 1709787326.52 total 490.90 last 12.20) - open/close 9283 (time 1709787336.52 total 500.91 last 15.50) - open/close 9503 (time 1709787346.53 total 510.91 last 21.99) - open/close 9639 (time 1709787357.42 total 521.81 last 12.49) - open/close 10000 (time 1709787365.11 total 529.49 last 46.95) - open/close 10230 (time 1709787380.17 total 544.55 last 15.28) - open/close 10766 (time 1709787390.17 total 554.55 last 53.59) - open/close 11318 (time 1709787400.18 total 564.56 last 55.15) - open/close 11811 (time 1709787410.18 total 574.56 last 49.29) - open/close 12311 (time 1709787420.18 total 584.57 last 50.00) - open/close 12843 (time 1709787430.21 total 594.60 last 53.04) - open/close 13342 (time 1709787440.21 total 604.60 last 49.89) - open/close 13804 (time 1709787461.06 total 625.45 last 22.16) - open/close 14146 (time 1709787471.06 total 635.45 last 34.19) - open/close 14600 (time 1709787481.07 total 645.45 last 45.39) - open/close 14992 (time 1709787491.13 total 655.52 last 38.94) - open/close 15373 (time 1709787501.18 total 665.57 last 37.92) - open/close 15840 (time 1709787511.23 total 675.61 last 46.49) - open/close 16272 (time 1709787521.24 total 685.62 last 43.16) - open/close 16638 (time 1709787531.26 total 695.64 last 36.52) - open/close 17058 (time 1709787541.27 total 705.65 last 41.95) - open/close 17461 (time 1709787551.28 total 715.67 last 40.24) - open/close 17781 (time 1709787561.29 total 725.68 last 31.97) - open/close 18139 (time 1709787571.30 total 735.68 last 35.79) - open/close 18463 (time 1709787581.32 total 745.70 last 32.34) - open/close 18816 (time 1709787591.35 total 755.74 last 35.17) - open/close 19139 (time 1709787601.36 total 765.74 last 32.29) - open/close 19484 (time 1709787611.38 total 775.76 last 34.43) - open/close 19831 (time 1709787621.40 total 785.78 last 34.62) - open/close 20000 (time 1709787626.68 total 791.06 last 32.03) - open/close 20334 (time 1709787636.71 total 801.09 last 33.30) - open/close 20649 (time 1709787646.72 total 811.11 last 31.46) - open/close 20991 (time 1709787656.72 total 821.11 last 34.19) - open/close 21324 (time 1709787666.72 total 831.11 last 33.30) - open/close 21649 (time 1709787676.73 total 841.12 last 32.48) - open/close 21961 (time 1709787686.75 total 851.14 last 31.13) - open/close 22244 (time 1709787696.78 total 861.17 last 28.21) - open/close 22495 (time 1709787706.81 total 871.19 last 25.03) - open/close 22801 (time 1709787727.70 total 892.09 last 14.65) - open/close 23108 (time 1709787737.73 total 902.11 last 30.63) - open/close 23401 (time 1709787747.73 total 912.11 last 29.29) - open/close 23685 (time 1709787757.75 total 922.14 last 28.34) - open/close 24000 (time 1709787767.77 total 932.15 last 31.45) - open/close 24287 (time 1709787777.79 total 942.17 last 28.64) - open/close 24600 (time 1709787787.82 total 952.20 last 31.22) - open/close 24921 (time 1709787797.84 total 962.23 last 32.02) - open/close 25245 (time 1709787807.86 total 972.25 last 32.33) - open/close 25572 (time 1709787817.89 total 982.28 last 32.60) - open/close 25892 (time 1709787827.91 total 992.30 last 31.93) - open/close 26212 (time 1709787837.92 total 1002.31 last 31.97) - open/close 26530 (time 1709787847.93 total 1012.32 last 31.77) - open/close 26881 (time 1709787857.96 total 1022.34 last 35.01) - open/close 27192 (time 1709787867.96 total 1032.34 last 31.10) - open/close 27473 (time 1709787877.97 total 1042.36 last 28.06) - open/close 27781 (time 1709787888.02 total 1052.40 last 30.66) - open/close 28071 (time 1709787898.04 total 1062.42 last 28.95) - open/close 28395 (time 1709787908.06 total 1072.44 last 32.33) - open/close 28715 (time 1709787918.10 total 1082.49 last 31.86) - open/close 29013 (time 1709787928.13 total 1092.52 last 29.72) - open/close 29285 (time 1709787938.17 total 1102.55 last 27.11) - open/close 29559 (time 1709787948.18 total 1112.56 last 27.37) - open/close 29867 (time 1709787958.18 total 1122.57 last 30.79) - open/close 30000 (time 1709787962.87 total 1127.25 last 28.37) - open/close 30309 (time 1709787972.88 total 1137.27 last 30.86) - open/close 30602 (time 1709787982.90 total 1147.28 last 29.26) - open/close 30920 (time 1709787992.90 total 1157.29 last 31.79) - open/close 31223 (time 1709788002.93 total 1167.31 last 30.22) - open/close 31522 (time 1709788012.97 total 1177.35 last 29.79) - open/close 31826 (time 1709788022.97 total 1187.36 last 30.38) - open/close 32107 (time 1709788033.00 total 1197.38 last 28.03) - open/close 32396 (time 1709788043.04 total 1207.42 last 28.78) - open/close 32673 (time 1709788053.07 total 1217.46 last 27.61) - open/close 32949 (time 1709788063.09 total 1227.48 last 27.55) - open/close 33249 (time 1709788073.10 total 1237.48 last 29.99) - open/close 33620 (time 1709788083.10 total 1247.49 last 37.07) - open/close 33968 (time 1709788093.11 total 1257.49 last 34.78) - open/close 34312 (time 1709788103.12 total 1267.51 last 34.35) - open/close 34648 (time 1709788113.14 total 1277.52 last 33.56) - open/close 34988 (time 1709788123.15 total 1287.54 last 33.95) - open/close 35325 (time 1709788133.16 total 1297.54 last 33.69) - open/close 35668 (time 1709788143.17 total 1307.55 last 34.26) - open/close 35988 (time 1709788153.17 total 1317.55 last 32.00) - open/close 36305 (time 1709788163.19 total 1327.57 last 31.64) - open/close 36636 (time 1709788173.20 total 1337.59 last 33.05) - open/close 36952 (time 1709788183.21 total 1347.59 last 31.57) - open/close 37265 (time 1709788193.23 total 1357.62 last 31.24) - open/close 37583 (time 1709788203.24 total 1367.63 last 31.76) - open/close 37897 (time 1709788213.24 total 1377.63 last 31.39) - open/close 38211 (time 1709788223.27 total 1387.66 last 31.31) - open/close 38520 (time 1709788233.29 total 1397.68 last 30.83) - open/close 38833 (time 1709788243.32 total 1407.71 last 31.21) - open/close 39125 (time 1709788253.33 total 1417.71 last 29.18) - open/close 39415 (time 1709788263.36 total 1427.74 last 28.92) - open/close 39712 (time 1709788273.36 total 1437.75 last 29.69) - open/close 40000 (time 1709788283.40 total 1447.79 last 28.68) - open/close 40261 (time 1709788293.41 total 1457.79 last 26.08) - open/close 40537 (time 1709788303.44 total 1467.82 last 27.52) - open/close 40816 (time 1709788313.46 total 1477.84 last 27.84) - open/close 41096 (time 1709788323.47 total 1487.86 last 27.96) - open/close 41387 (time 1709788333.48 total 1497.87 last 29.07) - open/close 41677 (time 1709788343.50 total 1507.88 last 28.96) - open/close 41964 (time 1709788353.52 total 1517.90 last 28.65) - open/close 42259 (time 1709788363.54 total 1527.92 last 29.44) - open/close 42520 (time 1709788373.55 total 1537.94 last 26.05) - open/close 42801 (time 1709788383.56 total 1547.94 last 28.09) - open/close 43072 (time 1709788393.56 total 1557.94 last 27.10) - open/close 43351 (time 1709788403.58 total 1567.97 last 27.83) - open/close 43625 (time 1709788413.61 total 1577.99 last 27.33) - open/close 43908 (time 1709788423.61 total 1588.00 last 28.29) - open/close 44185 (time 1709788433.63 total 1598.02 last 27.64) - open/close 44472 (time 1709788443.64 total 1608.02 last 28.69) - open/close 44749 (time 1709788453.67 total 1618.06 last 27.60) - open/close 45014 (time 1709788463.70 total 1628.09 last 26.43) - open/close 45289 (time 1709788473.72 total 1638.11 last 27.44) - open/close 45555 (time 1709788483.73 total 1648.11 last 26.58) - open/close 45807 (time 1709788493.75 total 1658.13 last 25.15) - open/close 46061 (time 1709788503.78 total 1668.17 last 25.31) - open/close 46321 (time 1709788513.78 total 1678.17 last 26.00) - open/close 46581 (time 1709788523.81 total 1688.20 last 25.92) - open/close 46836 (time 1709788533.83 total 1698.22 last 25.44) - open/close 47098 (time 1709788543.86 total 1708.25 last 26.12) - open/close 47353 (time 1709788553.88 total 1718.26 last 25.47) - open/close 47612 (time 1709788563.89 total 1728.28 last 25.86) - open/close 47875 (time 1709788573.91 total 1738.30 last 26.24) - open/close 48133 (time 1709788583.93 total 1748.31 last 25.76) - open/close 48392 (time 1709788593.94 total 1758.32 last 25.87) - open/close 48651 (time 1709788603.96 total 1768.35 last 25.84) - open/close 48909 (time 1709788613.99 total 1778.37 last 25.73) - open/close 49164 (time 1709788624.00 total 1788.39 last 25.46) - open/close 49411 (time 1709788634.03 total 1798.42 last 24.63) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1807.18 seconds: 27.45 ops/second - unlinked 0 (time 1709788644 ; total 0 ; last 0) - unlinked 10000 (time 1709789100 ; total 456 ; last 456) - unlinked 20000 (time 1709789471 ; total 827 ; last 371) - unlinked 30000 (time 1709789739 ; total 1095 ; last 268) - unlinked 40000 (time 1709789889 ; total 1245 ; last 150) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1297 seconds: 38.242096 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg333-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg333-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg333-server unloading modules on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2 PASS 101a (3160s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg333-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg333-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg333-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2 oleg333-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg333-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:39:25 (1709789965) start mds service on oleg333-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing load_modules_local oleg333-server: Loading modules from /home/green/git/lustre-release/lustre oleg333-server: detected 4 online CPUs by sysfs oleg333-server: Force libcfs to create 2 CPU partitions oleg333-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg333-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg333-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg333-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-OST0000 oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg333-client.virtnet: -o user_xattr,flock oleg333-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg333-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server start ost1 service on oleg333-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-OST0000 oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 6 sec oleg333-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b6333000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6333000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg333-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg333-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg333-server unloading modules on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2 PASS 101b (103s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg333-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg333-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2 oleg333-server: oleg333-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing load_modules_local oleg333-server: Loading modules from /home/green/git/lustre-release/lustre oleg333-server: detected 4 online CPUs by sysfs oleg333-server: Force libcfs to create 2 CPU partitions oleg333-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg333-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3864 sec ========== 00:41:33 (1709790093) === conf-sanity: start cleanup 00:41:33 (1709790093) === === conf-sanity: finish cleanup 00:41:33 (1709790093) ===