== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 13:33:25 (1721410405) Stopping clients: oleg219-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg219-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2 oleg219-server: oleg219-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing load_modules_local oleg219-server: Loading modules from /home/green/git/lustre-release/lustre oleg219-server: detected 4 online CPUs by sysfs oleg219-server: Force libcfs to create 2 CPU partitions oleg219-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg219-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing load_modules_local oleg219-server: Loading modules from /home/green/git/lustre-release/lustre oleg219-server: detected 4 online CPUs by sysfs oleg219-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg219-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg219-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg219-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg219-client.virtnet: -o user_xattr,flock oleg219-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 49667 inodes available. Want 100500. rc=0 - open/close 3621 (time 1721410468.74 total 10.00 last 362.09) - open/close 7532 (time 1721410478.74 total 20.00 last 390.96) total: 10000 open/close in 26.83 seconds: 372.79 ops/second - unlinked 0 (time 1721410486 ; total 0 ; last 0) total: 10000 unlinks in 9 seconds: 1111.111084 unlinks/second - open/close 3822 (time 1721410506.49 total 10.00 last 382.08) - open/close 7355 (time 1721410516.49 total 20.00 last 353.29) total: 10000 open/close in 26.99 seconds: 370.49 ops/second - unlinked 0 (time 1721410524 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second - open/close 3295 (time 1721410547.74 total 10.00 last 329.43) - open/close 6465 (time 1721410557.74 total 20.01 last 316.90) - open/close 9953 (time 1721410567.75 total 30.01 last 348.66) total: 10000 open/close in 30.18 seconds: 331.31 ops/second - unlinked 0 (time 1721410569 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second - open/close 3540 (time 1721410593.02 total 10.00 last 353.97) - open/close 6782 (time 1721410603.02 total 20.00 last 324.14) total: 10000 open/close in 29.14 seconds: 343.12 ops/second - unlinked 0 (time 1721410612 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second - open/close 3939 (time 1721410634.42 total 10.00 last 393.90) - open/close 7820 (time 1721410644.42 total 20.00 last 388.09) total: 10000 open/close in 23.98 seconds: 417.07 ops/second - unlinked 0 (time 1721410649 ; total 0 ; last 0) total: 10000 unlinks in 10 seconds: 1000.000000 unlinks/second - open/close 3692 (time 1721410669.93 total 10.00 last 369.17) - open/close 7265 (time 1721410679.94 total 20.00 last 357.18) total: 10000 open/close in 26.23 seconds: 381.26 ops/second - unlinked 0 (time 1721410687 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second - open/close 4009 (time 1721410710.69 total 10.00 last 400.87) - open/close 7837 (time 1721410720.69 total 20.00 last 382.70) total: 10000 open/close in 26.69 seconds: 374.73 ops/second - unlinked 0 (time 1721410728 ; total 0 ; last 0) total: 10000 unlinks in 10 seconds: 1000.000000 unlinks/second - open/close 3536 (time 1721410749.57 total 10.00 last 353.48) - open/close 7128 (time 1721410759.57 total 20.01 last 359.12) total: 10000 open/close in 27.42 seconds: 364.64 ops/second - unlinked 0 (time 1721410768 ; total 0 ; last 0) total: 10000 unlinks in 9 seconds: 1111.111084 unlinks/second - open/close 5312 (time 1721410788.13 total 10.00 last 531.19) total: 10000 open/close in 19.41 seconds: 515.32 ops/second - unlinked 0 (time 1721410798 ; total 0 ; last 0) total: 10000 unlinks in 10 seconds: 1000.000000 unlinks/second - open/close 3823 (time 1721410819.12 total 10.00 last 382.23) - open/close 8098 (time 1721410829.12 total 20.00 last 427.49) total: 10000 open/close in 25.00 seconds: 400.02 ops/second - unlinked 0 (time 1721410835 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second - open/close 3888 (time 1721410859.76 total 10.00 last 388.75) - open/close 7371 (time 1721410869.77 total 20.00 last 348.23) total: 10000 open/close in 26.03 seconds: 384.10 ops/second - unlinked 0 (time 1721410876 ; total 0 ; last 0) total: 10000 unlinks in 10 seconds: 1000.000000 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 4196 82396 5% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 95248 1504 85088 2% /mnt/lustre[MDT:1] lustre-OST0000_UUID 142216 1644 126572 2% /mnt/lustre[OST:0] filesystem_summary: 142216 1644 126572 2% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.110791 s, 9.5 MB/s fail_loc=0x80000204 Failing ost1 on oleg219-server Stopping /mnt/lustre-ost1 (opts:) on oleg219-server reboot facets: ost1 Failover ost1 to oleg219-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 100 open/close in 0.55 seconds: 181.59 ops/second umount lustre on /mnt/lustre..... Stopping client oleg219-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg219-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server stop mds service on oleg219-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server stop mds service on oleg219-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg219-server LNET ready to unload unloading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing unload_modules_local oleg219-server: LNET ready to unload modules unloaded.