== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 19:08:43 (1781046523) Stopping clients: oleg432-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg432-client.virtnet /mnt/lustre2 (opts:-f) oleg432-server: oleg432-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg432-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg432-server: detected 4 online CPUs by sysfs oleg432-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg432-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg432-server: detected 4 online CPUs by sysfs oleg432-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg432-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg432-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg432-client.virtnet: -o user_xattr,flock oleg432-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 2326 (time 1781046578.03 total 10.01 last 232.48) - open/close 4932 (time 1781046588.03 total 20.01 last 260.60) - open/close 7546 (time 1781046598.03 total 30.01 last 261.31) total: 10000 open/close in 38.04 seconds: 262.87 ops/second - unlinked 0 (time 1781046607 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 4080 (time 1781046636.85 total 10.00 last 407.98) - open/close 8676 (time 1781046646.85 total 20.00 last 459.52) total: 10000 open/close in 23.24 seconds: 430.28 ops/second - unlinked 0 (time 1781046651 ; total 0 ; last 0) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second - open/close 2265 (time 1781046678.08 total 10.00 last 226.47) - open/close 5481 (time 1781046688.08 total 20.00 last 321.55) - open/close 8489 (time 1781046698.09 total 30.01 last 300.69) total: 10000 open/close in 35.79 seconds: 279.44 ops/second - unlinked 0 (time 1781046705 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 4275 (time 1781046736.40 total 10.00 last 427.44) - open/close 8264 (time 1781046746.40 total 20.00 last 398.84) total: 10000 open/close in 24.65 seconds: 405.71 ops/second - unlinked 0 (time 1781046752 ; total 0 ; last 0) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second - open/close 4391 (time 1781046778.43 total 10.00 last 439.03) - open/close 8874 (time 1781046788.43 total 20.00 last 448.25) total: 10000 open/close in 22.95 seconds: 435.70 ops/second - unlinked 0 (time 1781046792 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 2829 (time 1781046821.10 total 10.00 last 282.90) - open/close 6294 (time 1781046831.10 total 20.00 last 346.47) total: 10000 open/close in 28.88 seconds: 346.27 ops/second - unlinked 0 (time 1781046841 ; total 0 ; last 0) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second - open/close 3582 (time 1781046867.67 total 10.00 last 358.17) - open/close 7010 (time 1781046877.67 total 20.00 last 342.80) total: 10000 open/close in 27.38 seconds: 365.21 ops/second - unlinked 0 (time 1781046886 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second - open/close 3804 (time 1781046912.03 total 10.00 last 380.34) - open/close 7770 (time 1781046922.03 total 20.00 last 396.55) total: 10000 open/close in 27.76 seconds: 360.26 ops/second - unlinked 0 (time 1781046931 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 4533 (time 1781046958.57 total 10.00 last 453.22) - open/close 9289 (time 1781046968.57 total 20.00 last 475.58) total: 10000 open/close in 21.31 seconds: 469.18 ops/second - unlinked 0 (time 1781046971 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second - open/close 4140 (time 1781046995.86 total 10.00 last 413.92) - open/close 8222 (time 1781047005.86 total 20.00 last 408.20) total: 10000 open/close in 24.32 seconds: 411.22 ops/second - unlinked 0 (time 1781047011 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second - open/close 4618 (time 1781047036.86 total 10.00 last 461.73) - open/close 9014 (time 1781047046.86 total 20.00 last 439.53) total: 10000 open/close in 22.12 seconds: 452.04 ops/second - unlinked 0 (time 1781047050 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 5504 376064 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 4096 376832 2% /mnt/lustre[OST:0] filesystem_summary: 382976 4096 376832 2% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131603 s, 8.0 MB/s fail_loc=0x80000204 Failing ost1 on oleg432-server Stopping /mnt/lustre-ost1 (opts:) on oleg432-server reboot facets: ost1 Failover ost1 to oleg432-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1781047089.14 total 2.59 last 0.00) total: 100 open/close in 3.18 seconds: 31.41 ops/second umount lustre on /mnt/lustre..... Stopping client oleg432-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg432-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg432-server stop mds service on oleg432-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg432-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing unload_modules_local oleg432-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg432-server: LNET ready to unload oleg432-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.