== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 09:17:25 (1759843045) Stopping clients: oleg332-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg332-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 oleg332-server: oleg332-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg332-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg332-server: detected 4 online CPUs by sysfs oleg332-server: Force libcfs to create 2 CPU partitions oleg332-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg332-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg332-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg332-server: detected 4 online CPUs by sysfs oleg332-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg332-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg332-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg332-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock oleg332-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 49667 inodes available. Want 100500. rc=0 - open/close 5236 (time 1759843108.84 total 10.00 last 523.52) total: 10000 open/close in 19.17 seconds: 521.72 ops/second - unlinked 0 (time 1759843119 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second - open/close 5394 (time 1759843141.45 total 10.00 last 539.36) total: 10000 open/close in 18.64 seconds: 536.40 ops/second - unlinked 0 (time 1759843151 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second - open/close 4689 (time 1759843174.90 total 10.00 last 468.87) - open/close 9127 (time 1759843184.90 total 20.00 last 443.78) total: 10000 open/close in 21.78 seconds: 459.06 ops/second - unlinked 0 (time 1759843188 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second - open/close 4659 (time 1759843211.77 total 10.00 last 465.83) - open/close 9477 (time 1759843221.77 total 20.00 last 481.78) total: 10000 open/close in 21.06 seconds: 474.82 ops/second - unlinked 0 (time 1759843224 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second - open/close 4556 (time 1759843247.84 total 10.00 last 455.51) - open/close 9141 (time 1759843257.84 total 20.00 last 458.48) total: 10000 open/close in 21.78 seconds: 459.23 ops/second - unlinked 0 (time 1759843260 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second - open/close 5685 (time 1759843283.27 total 10.00 last 568.45) total: 10000 open/close in 18.36 seconds: 544.65 ops/second - unlinked 0 (time 1759843292 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second - open/close 5114 (time 1759843314.99 total 10.00 last 511.39) total: 10000 open/close in 19.56 seconds: 511.29 ops/second - unlinked 0 (time 1759843325 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second - open/close 5277 (time 1759843348.06 total 10.00 last 527.66) total: 10000 open/close in 19.01 seconds: 526.06 ops/second - unlinked 0 (time 1759843358 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second - open/close 5756 (time 1759843380.54 total 10.00 last 575.58) total: 10000 open/close in 17.61 seconds: 567.79 ops/second - unlinked 0 (time 1759843389 ; total 0 ; last 0) total: 10000 unlinks in 10 seconds: 1000.000000 unlinks/second - open/close 5818 (time 1759843410.82 total 10.00 last 581.78) total: 10000 open/close in 17.03 seconds: 587.24 ops/second - unlinked 0 (time 1759843419 ; total 0 ; last 0) total: 10000 unlinks in 9 seconds: 1111.111084 unlinks/second - open/close 4837 (time 1759843440.23 total 10.00 last 483.65) - open/close 9878 (time 1759843450.23 total 20.00 last 504.02) total: 10000 open/close in 20.23 seconds: 494.31 ops/second - unlinked 0 (time 1759843451 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 4164 82428 5% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 95248 1504 85088 2% /mnt/lustre[MDT:1] lustre-OST0000_UUID 142216 1644 126572 2% /mnt/lustre[OST:0] filesystem_summary: 142216 1644 126572 2% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936881 s, 11.2 MB/s fail_loc=0x80000204 Failing ost1 on oleg332-server Stopping /mnt/lustre-ost1 (opts:) on oleg332-server reboot facets: ost1 Failover ost1 to oleg332-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 100 open/close in 0.74 seconds: 134.27 ops/second umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg332-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing unload_modules_local oleg332-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg332-server: LNET ready to unload oleg332-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.