-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:40:48 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg120-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg120-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg120-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg120-server oleg120-server: oleg120-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: detected 4 online CPUs by sysfs oleg120-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg120-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg120-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg120-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg120-server: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg120-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg120-server: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg120-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg120-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:42:28 (1709786548) start mds service on oleg120-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg120-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg120-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock oleg120-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h43m03s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff88012d7f6800.stats=clear mdc.lustre-MDT0001-mdc-ffff88012d7f6800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:43:47.000000000 -0500 Modify: 2024-03-06 23:43:47.000000000 -0500 Change: 2024-03-06 23:43:47.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 82 lines, 82 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg120-server unloading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing unload_modules_local modules unloaded. oleg120-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg120-server: Is the ldiskfs module available? oleg120-server: oleg120-server: tunefs.lustre FATAL: failed to write local files oleg120-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg120-client: oleg120-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg120-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg120-server: Is the ldiskfs module available? oleg120-server: oleg120-server: tunefs.lustre FATAL: failed to write local files oleg120-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg120-client: oleg120-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.120@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.120@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg120-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg120-server: Is the ldiskfs module available? oleg120-server: oleg120-server: tunefs.lustre FATAL: failed to write local files oleg120-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg120-client: oleg120-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.120@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.120@tcp sys.timeout=20 oleg120-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg120-server: Is the ldiskfs module available? oleg120-server: oleg120-server: tunefs.lustre FATAL: failed to write local files oleg120-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg120-client: oleg120-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.120@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.120@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg120-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg120-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 oleg120-server: oleg120-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: detected 4 online CPUs by sysfs oleg120-server: Force libcfs to create 2 CPU partitions oleg120-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg120-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg120-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg120-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg120-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg120-server: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg120-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg120-server: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg120-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg120-server PASS 35b (180s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:45:31 (1709786731) start mds service on oleg120-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg120-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg120-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock oleg120-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg120-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg120-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg120-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock oleg120-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg120-server 000000 2 000008 oleg120-server: 8+0 records in oleg120-server: 8+0 records out oleg120-server: 4096 bytes (4.1 kB) copied, 0.0010734 s, 3.8 MB/s start mds service on oleg120-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg120-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock oleg120-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg120-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server stop mds service on oleg120-server unloading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 PASS 38 (144s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg120-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg120-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg120-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 oleg120-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg120-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:47:59 (1709786879) Stopping clients: oleg120-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg120-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 oleg120-server: oleg120-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: detected 4 online CPUs by sysfs oleg120-server: Force libcfs to create 2 CPU partitions oleg120-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg120-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg120-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg120-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg120-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock oleg120-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg120-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg120-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg120-server umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg120-server unloading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing unload_modules_local modules unloaded. oleg120-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg120-server: Is the ldiskfs module available? oleg120-server: oleg120-server: tunefs.lustre FATAL: failed to write local files oleg120-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg120-client: oleg120-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg120-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg120-server: Is the ldiskfs module available? oleg120-server: oleg120-server: tunefs.lustre FATAL: failed to write local files oleg120-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg120-client: oleg120-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.120@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.120@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg120-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg120-server: Is the ldiskfs module available? oleg120-server: oleg120-server: tunefs.lustre FATAL: failed to write local files oleg120-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg120-client: oleg120-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.120@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.120@tcp sys.timeout=20 oleg120-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg120-server: Is the ldiskfs module available? oleg120-server: oleg120-server: tunefs.lustre FATAL: failed to write local files oleg120-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg120-client: oleg120-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.120@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.120@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg120-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg120-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 oleg120-server: oleg120-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: detected 4 online CPUs by sysfs oleg120-server: Force libcfs to create 2 CPU partitions oleg120-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg120-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg120-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg120-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg120-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg120-server: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg120-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec oleg120-server: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg120-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg120-server PASS 51 (251s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:52:14 (1709787134) start mds service on oleg120-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg120-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg120-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock oleg120-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 169 (time 1709787188.51 total 10.43 last 16.20) - open/close 372 (time 1709787199.42 total 21.34 last 18.60) - open/close 601 (time 1709787210.10 total 32.02 last 21.45) - open/close 810 (time 1709787220.52 total 42.44 last 20.05) - open/close 995 (time 1709787230.54 total 52.46 last 18.47) - open/close 1219 (time 1709787240.56 total 62.48 last 22.37) - open/close 1413 (time 1709787251.45 total 73.37 last 17.81) - open/close 1582 (time 1709787261.69 total 83.61 last 16.50) - open/close 1749 (time 1709787272.35 total 94.27 last 15.68) - open/close 2024 (time 1709787292.63 total 114.55 last 13.56) - open/close 2198 (time 1709787302.64 total 124.56 last 17.37) - open/close 2383 (time 1709787312.70 total 134.62 last 18.40) - open/close 2518 (time 1709787323.04 total 144.96 last 13.05) - open/close 2641 (time 1709787333.22 total 155.14 last 12.09) - open/close 2847 (time 1709787344.09 total 166.01 last 18.95) - open/close 3031 (time 1709787355.10 total 177.02 last 16.71) - open/close 3200 (time 1709787365.11 total 187.03 last 16.89) - open/close 3367 (time 1709787375.13 total 197.05 last 16.66) - open/close 3370 (time 1709787386.28 total 208.20 last 0.27) - open/close 3578 (time 1709787398.25 total 220.17 last 17.38) - open/close 3820 (time 1709787409.11 total 231.03 last 22.30) - open/close 3891 (time 1709787419.46 total 241.38 last 6.85) - open/close 4095 (time 1709787429.47 total 251.39 last 20.38) - open/close 4240 (time 1709787439.49 total 261.41 last 14.48) - open/close 4423 (time 1709787449.92 total 271.84 last 17.54) - open/close 4620 (time 1709787460.04 total 281.96 last 19.45) - open/close 4788 (time 1709787470.47 total 292.39 last 16.12) - open/close 4948 (time 1709787480.48 total 302.40 last 15.98) - open/close 5061 (time 1709787490.63 total 312.55 last 11.13) - open/close 5215 (time 1709787501.31 total 323.23 last 14.42) - open/close 5390 (time 1709787511.32 total 333.24 last 17.48) - open/close 5530 (time 1709787522.09 total 344.01 last 13.00) - open/close 5632 (time 1709787532.26 total 354.18 last 10.03) - open/close 5804 (time 1709787542.29 total 364.21 last 17.16) - open/close 5965 (time 1709787552.37 total 374.29 last 15.98) - open/close 6163 (time 1709787562.78 total 384.70 last 19.01) - open/close 6324 (time 1709787573.39 total 395.31 last 15.17) - open/close 6457 (time 1709787584.42 total 406.34 last 12.06) - open/close 6524 (time 1709787598.90 total 420.82 last 4.63) - open/close 6637 (time 1709787609.24 total 431.16 last 10.93) - open/close 6840 (time 1709787620.17 total 442.09 last 18.58) - open/close 7022 (time 1709787631.22 total 453.14 last 16.46) - open/close 7154 (time 1709787642.03 total 463.95 last 12.22) - open/close 7288 (time 1709787652.13 total 474.05 last 13.26) - open/close 7366 (time 1709787662.15 total 484.07 last 7.79) - open/close 7533 (time 1709787672.16 total 494.08 last 16.69) - open/close 7670 (time 1709787682.17 total 504.09 last 13.68) - open/close 7754 (time 1709787693.11 total 515.03 last 7.68) - open/close 7861 (time 1709787703.49 total 525.41 last 10.31) - open/close 8061 (time 1709787714.38 total 536.30 last 18.37) - open/close 8227 (time 1709787725.06 total 546.98 last 15.54) - open/close 8318 (time 1709787735.84 total 557.76 last 8.44) - open/close 8444 (time 1709787746.87 total 568.79 last 11.43) - open/close 8612 (time 1709787757.56 total 579.48 last 15.70) - open/close 9032 (time 1709787767.57 total 589.49 last 41.96) - open/close 9487 (time 1709787787.27 total 609.19 last 23.10) - open/close 9972 (time 1709787797.29 total 619.21 last 48.41) - open/close 10511 (time 1709787807.29 total 629.21 last 53.87) - open/close 11037 (time 1709787817.30 total 639.22 last 52.55) - open/close 11517 (time 1709787827.31 total 649.23 last 47.95) - open/close 11939 (time 1709787837.33 total 659.25 last 42.14) - open/close 12359 (time 1709787847.34 total 669.26 last 41.94) - open/close 12756 (time 1709787857.35 total 679.27 last 39.67) - open/close 13205 (time 1709787867.35 total 689.27 last 44.88) - open/close 13614 (time 1709787877.36 total 699.28 last 40.87) - open/close 14012 (time 1709787887.36 total 709.28 last 39.79) - open/close 14427 (time 1709787897.37 total 719.29 last 41.45) - open/close 14796 (time 1709787907.41 total 729.33 last 36.77) - open/close 15139 (time 1709787917.43 total 739.35 last 34.24) - open/close 15471 (time 1709787927.44 total 749.36 last 33.16) - open/close 15848 (time 1709787937.44 total 759.36 last 37.70) - open/close 16206 (time 1709787947.46 total 769.38 last 35.72) - open/close 16607 (time 1709787957.48 total 779.40 last 40.01) - open/close 16963 (time 1709787967.51 total 789.43 last 35.52) - open/close 17303 (time 1709787977.53 total 799.45 last 33.93) - open/close 17687 (time 1709787987.56 total 809.48 last 38.27) - open/close 18060 (time 1709787997.58 total 819.50 last 37.24) - open/close 18430 (time 1709788007.58 total 829.50 last 36.99) - open/close 18785 (time 1709788017.59 total 839.51 last 35.47) - open/close 19144 (time 1709788027.59 total 849.51 last 35.89) - open/close 19515 (time 1709788037.61 total 859.53 last 37.04) - open/close 19907 (time 1709788047.63 total 869.55 last 39.12) - open/close 20000 (time 1709788050.17 total 872.09 last 36.65) - open/close 20360 (time 1709788060.17 total 882.09 last 35.99) - open/close 20706 (time 1709788070.19 total 892.11 last 34.52) - open/close 21064 (time 1709788080.21 total 902.13 last 35.73) - open/close 21425 (time 1709788090.22 total 912.14 last 36.08) - open/close 21774 (time 1709788100.25 total 922.17 last 34.79) - open/close 22104 (time 1709788110.27 total 932.19 last 32.93) - open/close 22430 (time 1709788120.28 total 942.20 last 32.55) - open/close 22754 (time 1709788130.30 total 952.22 last 32.36) - open/close 23098 (time 1709788140.32 total 962.24 last 34.33) - open/close 23393 (time 1709788150.34 total 972.26 last 29.43) - open/close 23708 (time 1709788160.34 total 982.26 last 31.49) - open/close 24015 (time 1709788170.36 total 992.28 last 30.66) - open/close 24315 (time 1709788180.38 total 1002.30 last 29.92) - open/close 24622 (time 1709788190.39 total 1012.31 last 30.68) - open/close 24912 (time 1709788200.39 total 1022.31 last 29.00) - open/close 25242 (time 1709788210.40 total 1032.32 last 32.98) - open/close 25572 (time 1709788220.41 total 1042.33 last 32.96) - open/close 25912 (time 1709788230.43 total 1052.35 last 33.93) - open/close 26248 (time 1709788240.43 total 1062.35 last 33.58) - open/close 26555 (time 1709788250.46 total 1072.38 last 30.63) - open/close 26830 (time 1709788260.47 total 1082.39 last 27.47) - open/close 27119 (time 1709788270.47 total 1092.39 last 28.89) - open/close 27426 (time 1709788280.48 total 1102.40 last 30.67) - open/close 27749 (time 1709788290.49 total 1112.41 last 32.27) - open/close 28061 (time 1709788300.51 total 1122.43 last 31.12) - open/close 28363 (time 1709788310.54 total 1132.46 last 30.13) - open/close 28660 (time 1709788320.56 total 1142.48 last 29.64) - open/close 28961 (time 1709788330.58 total 1152.50 last 30.04) - open/close 29247 (time 1709788340.62 total 1162.54 last 28.49) - open/close 29528 (time 1709788350.62 total 1172.54 last 28.10) - open/close 29818 (time 1709788360.65 total 1182.57 last 28.90) - open/close 30000 (time 1709788366.61 total 1188.53 last 30.56) - open/close 30273 (time 1709788376.61 total 1198.53 last 27.29) - open/close 30546 (time 1709788386.63 total 1208.55 last 27.25) - open/close 30831 (time 1709788396.66 total 1218.58 last 28.41) - open/close 31106 (time 1709788406.69 total 1228.61 last 27.44) - open/close 31396 (time 1709788416.71 total 1238.63 last 28.94) - open/close 31653 (time 1709788426.75 total 1248.67 last 25.60) - open/close 31903 (time 1709788436.79 total 1258.71 last 24.90) - open/close 32139 (time 1709788446.79 total 1268.71 last 23.59) - open/close 32387 (time 1709788456.80 total 1278.72 last 24.79) - open/close 32615 (time 1709788466.84 total 1288.76 last 22.69) - open/close 32864 (time 1709788476.88 total 1298.80 last 24.82) - open/close 33108 (time 1709788486.88 total 1308.80 last 24.38) - open/close 33371 (time 1709788496.89 total 1318.81 last 26.28) - open/close 33639 (time 1709788506.90 total 1328.82 last 26.78) - open/close 33915 (time 1709788516.93 total 1338.85 last 27.51) - open/close 34173 (time 1709788526.93 total 1348.85 last 25.80) - open/close 34439 (time 1709788536.94 total 1358.86 last 26.57) - open/close 34701 (time 1709788546.97 total 1368.89 last 26.13) - open/close 34958 (time 1709788556.97 total 1378.89 last 25.69) - open/close 35221 (time 1709788566.98 total 1388.90 last 26.28) - open/close 35472 (time 1709788576.98 total 1398.90 last 25.09) - open/close 35698 (time 1709788587.02 total 1408.94 last 22.51) - open/close 35952 (time 1709788597.04 total 1418.96 last 25.36) - open/close 36206 (time 1709788607.04 total 1428.96 last 25.39) - open/close 36472 (time 1709788617.05 total 1438.97 last 26.56) - open/close 36752 (time 1709788627.06 total 1448.98 last 27.99) - open/close 37013 (time 1709788637.06 total 1458.98 last 26.08) - open/close 37265 (time 1709788647.08 total 1469.00 last 25.16) - open/close 37513 (time 1709788657.10 total 1479.02 last 24.74) - open/close 37758 (time 1709788667.12 total 1489.04 last 24.45) - open/close 37991 (time 1709788677.13 total 1499.05 last 23.29) - open/close 38205 (time 1709788687.13 total 1509.05 last 21.39) - open/close 38422 (time 1709788697.14 total 1519.06 last 21.69) - open/close 38662 (time 1709788707.17 total 1529.09 last 23.93) - open/close 38910 (time 1709788717.20 total 1539.12 last 24.72) - open/close 39149 (time 1709788727.22 total 1549.14 last 23.86) - open/close 39372 (time 1709788737.23 total 1559.15 last 22.27) - open/close 39585 (time 1709788747.25 total 1569.17 last 21.26) - open/close 39800 (time 1709788757.29 total 1579.21 last 21.41) - open/close 40000 (time 1709788766.76 total 1588.68 last 21.11) - open/close 40203 (time 1709788776.78 total 1598.70 last 20.26) - open/close 40425 (time 1709788786.81 total 1608.73 last 22.14) - open/close 40666 (time 1709788796.84 total 1618.76 last 24.02) - open/close 40899 (time 1709788806.85 total 1628.77 last 23.28) - open/close 41139 (time 1709788816.88 total 1638.80 last 23.93) - open/close 41366 (time 1709788826.91 total 1648.83 last 22.62) - open/close 41604 (time 1709788836.93 total 1658.85 last 23.76) - open/close 41841 (time 1709788846.96 total 1668.88 last 23.61) - open/close 42073 (time 1709788856.99 total 1678.91 last 23.14) - open/close 42297 (time 1709788867.00 total 1688.92 last 22.37) - open/close 42538 (time 1709788877.04 total 1698.96 last 24.01) - open/close 42769 (time 1709788887.07 total 1708.99 last 23.03) - open/close 43008 (time 1709788897.09 total 1719.01 last 23.87) - open/close 43225 (time 1709788907.10 total 1729.02 last 21.68) - open/close 43450 (time 1709788917.12 total 1739.04 last 22.44) - open/close 43664 (time 1709788927.15 total 1749.07 last 21.35) - open/close 43887 (time 1709788937.19 total 1759.11 last 22.21) - open/close 44094 (time 1709788947.22 total 1769.14 last 20.64) - open/close 44299 (time 1709788957.25 total 1779.17 last 20.43) - open/close 44502 (time 1709788967.28 total 1789.20 last 20.25) - open/close 44711 (time 1709788977.28 total 1799.20 last 20.89) - open/close 44910 (time 1709788987.32 total 1809.24 last 19.82) - open/close 45118 (time 1709788997.33 total 1819.25 last 20.78) - open/close 45313 (time 1709789007.36 total 1829.28 last 19.45) - open/close 45513 (time 1709789017.39 total 1839.31 last 19.94) - open/close 45727 (time 1709789027.43 total 1849.35 last 21.31) - open/close 45938 (time 1709789037.44 total 1859.36 last 21.08) - open/close 46134 (time 1709789047.45 total 1869.37 last 19.57) - open/close 46331 (time 1709789057.45 total 1879.37 last 19.70) - open/close 46529 (time 1709789067.47 total 1889.39 last 19.76) - open/close 46724 (time 1709789077.51 total 1899.43 last 19.42) - open/close 46902 (time 1709789087.54 total 1909.46 last 17.73) - open/close 47094 (time 1709789097.55 total 1919.47 last 19.18) - open/close 47289 (time 1709789107.59 total 1929.51 last 19.44) - open/close 47480 (time 1709789117.61 total 1939.53 last 19.06) - open/close 47660 (time 1709789127.63 total 1949.55 last 17.95) - open/close 47841 (time 1709789137.68 total 1959.60 last 18.02) - open/close 48028 (time 1709789147.68 total 1969.60 last 18.69) - open/close 48218 (time 1709789157.69 total 1979.61 last 18.99) - open/close 48403 (time 1709789167.70 total 1989.62 last 18.47) - open/close 48586 (time 1709789177.70 total 1999.62 last 18.30) - open/close 48759 (time 1709789187.72 total 2009.64 last 17.27) - open/close 48946 (time 1709789197.75 total 2019.67 last 18.65) - open/close 49130 (time 1709789207.80 total 2029.72 last 18.32) - open/close 49315 (time 1709789217.84 total 2039.76 last 18.42) - open/close 49508 (time 1709789227.87 total 2049.79 last 19.24) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 2054.61 seconds: 24.14 ops/second - unlinked 0 (time 1709789234 ; total 0 ; last 0) - unlinked 10000 (time 1709789790 ; total 556 ; last 556) - unlinked 20000 (time 1709790210 ; total 976 ; last 420) - unlinked 30000 (time 1709790433 ; total 1199 ; last 223) - unlinked 40000 (time 1709790550 ; total 1316 ; last 117) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1358 seconds: 36.524300 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg120-server unloading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 PASS 101a (3480s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg120-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg120-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg120-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 oleg120-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg120-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:50:15 (1709790615) start mds service on oleg120-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: detected 4 online CPUs by sysfs oleg120-server: Force libcfs to create 2 CPU partitions oleg120-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg120-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg120-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg120-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock oleg120-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server start ost1 service on oleg120-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec oleg120-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b6d26800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6d26800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg120-server unloading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 PASS 101b (88s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg120-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg120-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg120-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg120-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 oleg120-server: oleg120-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: detected 4 online CPUs by sysfs oleg120-server: Force libcfs to create 2 CPU partitions oleg120-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg120-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg120-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 4276 sec ========== 00:52:05 (1709790725) === conf-sanity: start cleanup 00:52:05 (1709790725) === === conf-sanity: finish cleanup 00:52:05 (1709790725) ===