-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:39:32 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg134-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg134-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg134-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg134-server oleg134-server: oleg134-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing load_modules_local oleg134-server: Loading modules from /home/green/git/lustre-release/lustre oleg134-server: detected 4 online CPUs by sysfs oleg134-server: Force libcfs to create 2 CPU partitions oleg134-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg134-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg134-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg134-server: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg134-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:40:23 (1709786423) start mds service on oleg134-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg134-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock oleg134-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h40m37s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff88012bf12000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:41:20.000000000 -0500 Modify: 2024-03-06 23:41:20.000000000 -0500 Change: 2024-03-06 23:41:20.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 56 lines, 56 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server unloading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing unload_modules_local modules unloaded. oleg134-server: oleg134-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg134-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg134-client: oleg134-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg134-server: oleg134-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg134-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg134-client: oleg134-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg134-server: oleg134-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg134-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg134-client: oleg134-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg134-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg134-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 oleg134-server: oleg134-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing load_modules_local oleg134-server: Loading modules from /home/green/git/lustre-release/lustre oleg134-server: detected 4 online CPUs by sysfs oleg134-server: Force libcfs to create 2 CPU partitions oleg134-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg134-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg134-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg134-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg134-server: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg134-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server PASS 35b (120s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:42:25 (1709786545) start mds service on oleg134-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg134-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock oleg134-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg134-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock oleg134-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:) stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server 000000 2 000008 oleg134-server: 8+0 records in oleg134-server: 8+0 records out oleg134-server: 4096 bytes (4.1 kB) copied, 0.00057293 s, 7.1 MB/s start mds service on oleg134-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock oleg134-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:) stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server stop mds service on oleg134-server unloading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 PASS 38 (70s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg134-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg134-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg134-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 oleg134-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg134-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:43:36 (1709786616) Stopping clients: oleg134-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg134-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 oleg134-server: oleg134-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing load_modules_local oleg134-server: Loading modules from /home/green/git/lustre-release/lustre oleg134-server: detected 4 online CPUs by sysfs oleg134-server: Force libcfs to create 2 CPU partitions oleg134-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg134-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg134-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg134-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg134-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock oleg134-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg134-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec stop ost2 service on oleg134-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg134-server umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server unloading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing unload_modules_local modules unloaded. oleg134-server: oleg134-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg134-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg134-client: oleg134-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg134-server: oleg134-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg134-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg134-client: oleg134-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg134-server: oleg134-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg134-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg134-client: oleg134-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg134-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg134-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 oleg134-server: oleg134-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing load_modules_local oleg134-server: Loading modules from /home/green/git/lustre-release/lustre oleg134-server: detected 4 online CPUs by sysfs oleg134-server: Force libcfs to create 2 CPU partitions oleg134-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg134-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg134-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg134-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg134-server: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg134-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server PASS 51 (159s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:46:18 (1709786778) start mds service on oleg134-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg134-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock oleg134-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 252 (time 1709786808.95 total 10.66 last 23.64) - open/close 495 (time 1709786819.74 total 21.45 last 22.53) - open/close 865 (time 1709786829.74 total 31.45 last 36.99) - open/close 971 (time 1709786840.26 total 41.96 last 10.08) - open/close 1208 (time 1709786850.70 total 52.40 last 22.70) - open/close 1354 (time 1709786862.15 total 63.86 last 12.75) - open/close 1568 (time 1709786872.89 total 74.60 last 19.91) - open/close 1723 (time 1709786882.93 total 84.64 last 15.44) - open/close 1951 (time 1709786893.40 total 95.11 last 21.78) - open/close 2153 (time 1709786904.15 total 105.86 last 18.80) - open/close 2433 (time 1709786916.02 total 117.72 last 23.59) - open/close 2643 (time 1709786928.83 total 130.54 last 16.39) - open/close 2880 (time 1709786939.30 total 141.01 last 22.63) - open/close 3034 (time 1709786949.71 total 151.42 last 14.80) - open/close 3354 (time 1709786960.12 total 161.83 last 30.73) - open/close 3628 (time 1709786970.87 total 172.58 last 25.49) - open/close 3933 (time 1709786980.87 total 182.58 last 30.50) - open/close 4230 (time 1709786991.52 total 193.23 last 27.90) - open/close 4387 (time 1709787002.45 total 204.16 last 14.36) - open/close 4520 (time 1709787012.46 total 214.17 last 13.29) - open/close 4806 (time 1709787022.88 total 224.58 last 27.45) - open/close 5105 (time 1709787032.90 total 234.60 last 29.85) - open/close 5363 (time 1709787043.70 total 245.41 last 23.87) - open/close 5563 (time 1709787054.05 total 255.76 last 19.33) - open/close 5730 (time 1709787064.05 total 265.76 last 16.70) - open/close 5951 (time 1709787074.07 total 275.78 last 22.06) - open/close 6152 (time 1709787084.30 total 286.00 last 19.66) - open/close 6399 (time 1709787094.73 total 296.44 last 23.67) - open/close 6592 (time 1709787105.37 total 307.08 last 18.14) - open/close 6838 (time 1709787115.56 total 317.27 last 24.14) - open/close 6982 (time 1709787125.59 total 327.30 last 14.36) - open/close 7145 (time 1709787135.63 total 337.34 last 16.24) - open/close 7281 (time 1709787145.63 total 347.34 last 13.59) - open/close 7455 (time 1709787155.64 total 357.35 last 17.39) - open/close 7591 (time 1709787166.27 total 367.98 last 12.79) - open/close 7830 (time 1709787177.25 total 378.96 last 21.76) - open/close 7957 (time 1709787187.53 total 389.24 last 12.35) - open/close 8186 (time 1709787197.63 total 399.33 last 22.69) - open/close 8347 (time 1709787207.63 total 409.34 last 16.09) - open/close 8522 (time 1709787217.83 total 419.54 last 17.16) - open/close 8658 (time 1709787228.67 total 430.38 last 12.55) - open/close 8805 (time 1709787238.69 total 440.40 last 14.67) - open/close 8944 (time 1709787248.71 total 450.41 last 13.88) - open/close 9102 (time 1709787259.34 total 461.05 last 14.86) - open/close 9426 (time 1709787269.35 total 471.06 last 32.35) - open/close 9609 (time 1709787280.11 total 481.82 last 17.02) - open/close 9806 (time 1709787290.94 total 492.65 last 18.19) - open/close 9983 (time 1709787301.79 total 503.50 last 16.31) - open/close 10113 (time 1709787311.79 total 513.50 last 13.00) - open/close 10261 (time 1709787321.81 total 523.52 last 14.77) - open/close 10487 (time 1709787331.81 total 533.52 last 22.60) - open/close 10941 (time 1709787341.82 total 543.52 last 45.38) - open/close 11460 (time 1709787351.83 total 553.54 last 51.82) - open/close 11985 (time 1709787361.86 total 563.57 last 52.35) - open/close 12383 (time 1709787380.80 total 582.50 last 21.02) - open/close 12829 (time 1709787390.84 total 592.54 last 44.43) - open/close 13267 (time 1709787400.86 total 602.57 last 43.70) - open/close 13694 (time 1709787410.87 total 612.58 last 42.65) - open/close 14153 (time 1709787420.90 total 622.60 last 45.78) - open/close 14609 (time 1709787430.91 total 632.61 last 45.56) - open/close 15045 (time 1709787440.92 total 642.62 last 43.55) - open/close 15431 (time 1709787450.94 total 652.65 last 38.51) - open/close 15838 (time 1709787460.94 total 662.65 last 40.68) - open/close 16243 (time 1709787470.97 total 672.68 last 40.40) - open/close 16633 (time 1709787480.98 total 682.69 last 38.94) - open/close 17004 (time 1709787491.00 total 692.71 last 37.05) - open/close 17379 (time 1709787501.01 total 702.72 last 37.46) - open/close 17725 (time 1709787511.02 total 712.72 last 34.58) - open/close 18086 (time 1709787521.08 total 722.79 last 35.86) - open/close 18460 (time 1709787531.08 total 732.79 last 37.40) - open/close 18792 (time 1709787551.27 total 752.98 last 16.44) - open/close 19118 (time 1709787561.30 total 763.01 last 32.51) - open/close 19448 (time 1709787571.33 total 773.03 last 32.92) - open/close 19800 (time 1709787581.33 total 783.04 last 35.19) - open/close 20000 (time 1709787586.87 total 788.58 last 36.08) - open/close 20365 (time 1709787596.87 total 798.58 last 36.49) - open/close 20731 (time 1709787606.89 total 808.60 last 36.55) - open/close 21098 (time 1709787616.91 total 818.62 last 36.62) - open/close 21466 (time 1709787626.93 total 828.64 last 36.71) - open/close 21811 (time 1709787636.94 total 838.65 last 34.47) - open/close 22163 (time 1709787646.95 total 848.66 last 35.18) - open/close 22468 (time 1709787657.02 total 858.73 last 30.29) - open/close 22770 (time 1709787667.05 total 868.75 last 30.12) - open/close 23071 (time 1709787677.06 total 878.77 last 30.05) - open/close 23396 (time 1709787687.08 total 888.79 last 32.44) - open/close 23729 (time 1709787697.09 total 898.80 last 33.26) - open/close 24068 (time 1709787707.09 total 908.80 last 33.90) - open/close 24350 (time 1709787726.25 total 927.95 last 14.72) - open/close 24683 (time 1709787736.30 total 938.01 last 33.13) - open/close 24996 (time 1709787746.32 total 948.03 last 31.23) - open/close 25298 (time 1709787756.33 total 958.03 last 30.18) - open/close 25610 (time 1709787766.34 total 968.05 last 31.15) - open/close 25918 (time 1709787776.37 total 978.07 last 30.73) - open/close 26212 (time 1709787786.38 total 988.09 last 29.34) - open/close 26487 (time 1709787796.41 total 998.11 last 27.44) - open/close 26778 (time 1709787806.41 total 1008.12 last 29.07) - open/close 27099 (time 1709787816.44 total 1018.15 last 32.01) - open/close 27382 (time 1709787826.48 total 1028.19 last 28.19) - open/close 27659 (time 1709787836.50 total 1038.21 last 27.66) - open/close 27903 (time 1709787846.53 total 1048.24 last 24.32) - open/close 28028 (time 1709787862.57 total 1064.28 last 7.79) - open/close 28306 (time 1709787872.61 total 1074.32 last 27.69) - open/close 28583 (time 1709787882.63 total 1084.34 last 27.63) - open/close 28858 (time 1709787892.64 total 1094.34 last 27.49) - open/close 29134 (time 1709787902.64 total 1104.35 last 27.60) - open/close 29407 (time 1709787912.65 total 1114.36 last 27.27) - open/close 29679 (time 1709787922.67 total 1124.38 last 27.15) - open/close 29916 (time 1709787932.70 total 1134.41 last 23.62) - open/close 30000 (time 1709787935.83 total 1137.54 last 26.86) - open/close 30262 (time 1709787945.85 total 1147.56 last 26.14) - open/close 30559 (time 1709787955.85 total 1157.56 last 29.70) - open/close 30851 (time 1709787965.87 total 1167.57 last 29.16) - open/close 31144 (time 1709787975.87 total 1177.58 last 29.28) - open/close 31438 (time 1709787985.90 total 1187.61 last 29.31) - open/close 31733 (time 1709787995.92 total 1197.62 last 29.47) - open/close 32005 (time 1709788005.94 total 1207.65 last 27.12) - open/close 32303 (time 1709788015.98 total 1217.68 last 29.71) - open/close 32569 (time 1709788025.99 total 1227.70 last 26.57) - open/close 32839 (time 1709788035.99 total 1237.70 last 26.99) - open/close 33115 (time 1709788046.02 total 1247.72 last 27.53) - open/close 33381 (time 1709788056.03 total 1257.74 last 26.57) - open/close 33653 (time 1709788066.05 total 1267.76 last 27.13) - open/close 33929 (time 1709788076.06 total 1277.77 last 27.58) - open/close 34203 (time 1709788086.08 total 1287.78 last 27.35) - open/close 34491 (time 1709788096.10 total 1297.80 last 28.74) - open/close 34759 (time 1709788106.12 total 1307.83 last 26.74) - open/close 35026 (time 1709788116.13 total 1317.84 last 26.68) - open/close 35271 (time 1709788126.15 total 1327.86 last 24.45) - open/close 35537 (time 1709788136.17 total 1337.88 last 26.54) - open/close 35791 (time 1709788146.19 total 1347.90 last 25.34) - open/close 36038 (time 1709788156.22 total 1357.93 last 24.63) - open/close 36285 (time 1709788166.24 total 1367.95 last 24.66) - open/close 36537 (time 1709788176.27 total 1377.98 last 25.13) - open/close 36794 (time 1709788186.28 total 1387.98 last 25.68) - open/close 37045 (time 1709788196.29 total 1398.00 last 25.07) - open/close 37328 (time 1709788206.30 total 1408.01 last 28.27) - open/close 37609 (time 1709788216.31 total 1418.02 last 28.07) - open/close 37872 (time 1709788226.33 total 1428.04 last 26.25) - open/close 38152 (time 1709788236.33 total 1438.04 last 27.99) - open/close 38394 (time 1709788246.39 total 1448.10 last 24.07) - open/close 38628 (time 1709788256.43 total 1458.14 last 23.31) - open/close 38866 (time 1709788266.44 total 1468.15 last 23.77) - open/close 39104 (time 1709788276.45 total 1478.15 last 23.79) - open/close 39359 (time 1709788286.45 total 1488.16 last 25.48) - open/close 39613 (time 1709788296.48 total 1498.18 last 25.34) - open/close 39858 (time 1709788306.48 total 1508.19 last 24.49) - open/close 40000 (time 1709788312.10 total 1513.81 last 25.27) - open/close 40240 (time 1709788322.12 total 1523.82 last 23.96) - open/close 40493 (time 1709788332.15 total 1533.86 last 25.22) - open/close 40735 (time 1709788342.17 total 1543.87 last 24.16) - open/close 40975 (time 1709788352.19 total 1553.90 last 23.95) - open/close 41229 (time 1709788362.23 total 1563.94 last 25.29) - open/close 41465 (time 1709788372.27 total 1573.98 last 23.50) - open/close 41710 (time 1709788382.30 total 1584.01 last 24.44) - open/close 41956 (time 1709788392.32 total 1594.03 last 24.54) - open/close 42196 (time 1709788402.37 total 1604.08 last 23.88) - open/close 42442 (time 1709788412.39 total 1614.10 last 24.55) - open/close 42681 (time 1709788422.42 total 1624.13 last 23.83) - open/close 42910 (time 1709788432.43 total 1634.13 last 22.88) - open/close 43126 (time 1709788442.45 total 1644.16 last 21.54) - open/close 43338 (time 1709788452.45 total 1654.16 last 21.20) - open/close 43555 (time 1709788462.49 total 1664.20 last 21.62) - open/close 43784 (time 1709788472.51 total 1674.22 last 22.85) - open/close 44008 (time 1709788482.51 total 1684.22 last 22.40) - open/close 44238 (time 1709788492.54 total 1694.25 last 22.93) - open/close 44477 (time 1709788502.56 total 1704.26 last 23.86) - open/close 44715 (time 1709788512.56 total 1714.27 last 23.78) - open/close 44944 (time 1709788522.59 total 1724.30 last 22.83) - open/close 45179 (time 1709788532.62 total 1734.33 last 23.44) - open/close 45422 (time 1709788542.63 total 1744.34 last 24.27) - open/close 45659 (time 1709788552.67 total 1754.38 last 23.62) - open/close 45897 (time 1709788562.69 total 1764.40 last 23.76) - open/close 46138 (time 1709788572.72 total 1774.42 last 24.03) - open/close 46354 (time 1709788582.76 total 1784.47 last 21.50) - open/close 46585 (time 1709788592.76 total 1794.47 last 23.10) - open/close 46817 (time 1709788602.78 total 1804.49 last 23.15) - open/close 47051 (time 1709788612.80 total 1814.51 last 23.35) - open/close 47287 (time 1709788622.84 total 1824.55 last 23.52) - open/close 47521 (time 1709788632.86 total 1834.56 last 23.36) - open/close 47740 (time 1709788642.89 total 1844.60 last 21.82) - open/close 47959 (time 1709788652.93 total 1854.64 last 21.82) - open/close 48178 (time 1709788662.96 total 1864.67 last 21.83) - open/close 48394 (time 1709788672.96 total 1874.67 last 21.60) - open/close 48590 (time 1709788683.00 total 1884.70 last 19.54) - open/close 48805 (time 1709788693.00 total 1894.71 last 21.49) - open/close 49007 (time 1709788703.02 total 1904.73 last 20.15) - open/close 49223 (time 1709788713.06 total 1914.77 last 21.53) - open/close 49435 (time 1709788723.07 total 1924.77 last 21.18) - open/close 49649 (time 1709788733.10 total 1934.81 last 21.32) - open/close 49845 (time 1709788743.14 total 1944.85 last 19.53) total: 50000 open/close in 1952.70 seconds: 25.61 ops/second