== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 21:36:12 (1767839772) Stopping clients: oleg351-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:-f) oleg351-server: oleg351-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg351-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: detected 4 online CPUs by sysfs oleg351-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg351-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg351-server: detected 4 online CPUs by sysfs oleg351-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg351-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg351-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg351-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg351-client.virtnet: -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 1172 (time 1767839876.58 total 10.00 last 117.14) - open/close 2233 (time 1767839886.59 total 20.01 last 106.07) - open/close 3433 (time 1767839896.59 total 30.01 last 119.95) - open/close 4369 (time 1767839906.59 total 40.01 last 93.60) - open/close 5389 (time 1767839916.59 total 50.01 last 101.98) - open/close 6428 (time 1767839926.60 total 60.02 last 103.83) - open/close 7417 (time 1767839936.61 total 70.03 last 98.78) - open/close 8464 (time 1767839946.62 total 80.04 last 104.65) - open/close 9682 (time 1767839956.62 total 90.04 last 121.74) total: 10000 open/close in 92.92 seconds: 107.62 ops/second - unlinked 0 (time 1767839961 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1460 (time 1767840015.71 total 10.00 last 145.95) - open/close 2834 (time 1767840025.71 total 20.01 last 137.33) - open/close 4671 (time 1767840035.72 total 30.01 last 183.66) - open/close 6405 (time 1767840045.72 total 40.02 last 173.31) - open/close 7842 (time 1767840055.74 total 50.03 last 143.51) - open/close 9280 (time 1767840065.74 total 60.03 last 143.78) total: 10000 open/close in 64.42 seconds: 155.24 ops/second - unlinked 0 (time 1767840072 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 1810 (time 1767840113.21 total 10.01 last 180.90) - open/close 4128 (time 1767840123.21 total 20.01 last 231.80) - open/close 5671 (time 1767840133.21 total 30.01 last 154.26) - open/close 7076 (time 1767840143.22 total 40.01 last 140.41) - open/close 8222 (time 1767840153.22 total 50.02 last 114.53) - open/close 9469 (time 1767840163.23 total 60.03 last 124.62) total: 10000 open/close in 64.18 seconds: 155.81 ops/second - unlinked 0 (time 1767840169 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1988 (time 1767840217.22 total 10.00 last 198.72) - open/close 4159 (time 1767840227.22 total 20.01 last 217.06) - open/close 5740 (time 1767840237.22 total 30.01 last 158.09) - open/close 7305 (time 1767840247.23 total 40.01 last 156.44) - open/close 8879 (time 1767840257.23 total 50.01 last 157.38) total: 10000 open/close in 56.31 seconds: 177.58 ops/second - unlinked 0 (time 1767840265 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 2380 (time 1767840299.25 total 10.00 last 237.96) - open/close 4909 (time 1767840309.25 total 20.00 last 252.85) - open/close 7350 (time 1767840319.25 total 30.00 last 244.08) - open/close 9154 (time 1767840329.26 total 40.01 last 180.32) total: 10000 open/close in 44.08 seconds: 226.87 ops/second - unlinked 0 (time 1767840335 ; total 0 ; last 0) total: 10000 unlinks in 28 seconds: 357.142853 unlinks/second - open/close 1752 (time 1767840375.23 total 10.00 last 175.15) - open/close 3985 (time 1767840385.23 total 20.00 last 223.27) - open/close 7732 (time 1767840395.23 total 30.00 last 374.68) total: 10000 open/close in 35.33 seconds: 283.07 ops/second - unlinked 0 (time 1767840401 ; total 0 ; last 0) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second - open/close 3856 (time 1767840428.04 total 10.00 last 385.59) - open/close 8211 (time 1767840438.04 total 20.00 last 435.42) total: 10000 open/close in 23.66 seconds: 422.72 ops/second - unlinked 0 (time 1767840443 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second - open/close 4442 (time 1767840468.41 total 10.00 last 444.16) - open/close 9052 (time 1767840478.41 total 20.00 last 460.99) total: 10000 open/close in 21.87 seconds: 457.24 ops/second - unlinked 0 (time 1767840481 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second - open/close 4957 (time 1767840504.89 total 10.00 last 495.66) - open/close 9764 (time 1767840514.89 total 20.00 last 480.67) total: 10000 open/close in 20.49 seconds: 488.04 ops/second - unlinked 0 (time 1767840516 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second - open/close 4604 (time 1767840541.35 total 10.00 last 460.35) - open/close 9084 (time 1767840551.35 total 20.00 last 447.98) total: 10000 open/close in 22.10 seconds: 452.55 ops/second - unlinked 0 (time 1767840554 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second - open/close 4497 (time 1767840580.80 total 10.00 last 449.66) - open/close 9189 (time 1767840590.80 total 20.00 last 469.17) total: 10000 open/close in 21.75 seconds: 459.87 ops/second - unlinked 0 (time 1767840593 ; total 0 ; last 0) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4224 377344 2% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 383616 3072 378496 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 382976 4096 376832 2% /mnt/lustre[OST:0] filesystem_summary: 382976 4096 376832 2% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122534 s, 8.6 MB/s fail_loc=0x80000204 Failing ost1 on oleg351-server Stopping /mnt/lustre-ost1 (opts:) on oleg351-server reboot facets: ost1 Failover ost1 to oleg351-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1767840638.23 total 4.68 last 0.00) total: 100 open/close in 5.35 seconds: 18.70 ops/second umount lustre on /mnt/lustre..... Stopping client oleg351-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg351-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing unload_modules_local oleg351-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg351-server: LNET ready to unload oleg351-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.