== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 17:28:16 (1778534896) Stopping clients: oleg459-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg459-client.virtnet /mnt/lustre2 (opts:-f) oleg459-server: oleg459-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing load_modules_local oleg459-server: Loading modules from /home/green/git/lustre-release/lustre oleg459-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg459-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg459-server: detected 4 online CPUs by sysfs oleg459-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing load_modules_local oleg459-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg459-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg459-server: Loading modules from /home/green/git/lustre-release/lustre oleg459-server: detected 4 online CPUs by sysfs oleg459-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg459-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg459-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg459-client.virtnet: -o user_xattr,flock oleg459-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 944 (time 1778534984.15 total 10.02 last 94.24) - open/close 1892 (time 1778534994.16 total 20.02 last 94.75) - open/close 2868 (time 1778535004.16 total 30.03 last 97.54) - open/close 3925 (time 1778535014.17 total 40.04 last 105.60) - open/close 5080 (time 1778535024.18 total 50.04 last 115.45) - open/close 6127 (time 1778535034.19 total 60.05 last 104.63) - open/close 7094 (time 1778535044.19 total 70.05 last 96.66) - open/close 8066 (time 1778535054.19 total 80.05 last 97.19) - open/close 9037 (time 1778535064.19 total 90.06 last 97.05) - open/close 9913 (time 1778535074.20 total 100.07 last 87.54) total: 10000 open/close in 100.85 seconds: 99.16 ops/second - unlinked 0 (time 1778535077 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 1007 (time 1778535146.65 total 10.00 last 100.66) - open/close 2084 (time 1778535156.66 total 20.01 last 107.61) - open/close 3211 (time 1778535166.67 total 30.02 last 112.63) - open/close 4342 (time 1778535176.67 total 40.02 last 113.04) - open/close 5516 (time 1778535186.68 total 50.03 last 117.37) - open/close 6575 (time 1778535196.68 total 60.03 last 105.89) - open/close 7845 (time 1778535206.68 total 70.03 last 126.93) - open/close 9027 (time 1778535216.68 total 80.04 last 118.16) total: 10000 open/close in 86.81 seconds: 115.19 ops/second - unlinked 0 (time 1778535225 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1528 (time 1778535271.11 total 10.00 last 152.73) - open/close 3030 (time 1778535281.11 total 20.01 last 150.11) - open/close 4406 (time 1778535291.12 total 30.02 last 137.52) - open/close 5570 (time 1778535301.12 total 40.02 last 116.34) - open/close 6725 (time 1778535311.13 total 50.02 last 115.47) - open/close 7874 (time 1778535321.13 total 60.03 last 114.85) - open/close 8947 (time 1778535331.14 total 70.03 last 107.24) total: 10000 open/close in 80.00 seconds: 124.99 ops/second - unlinked 0 (time 1778535343 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1237 (time 1778535399.96 total 10.01 last 123.62) - open/close 2506 (time 1778535409.96 total 20.01 last 126.84) - open/close 3956 (time 1778535419.97 total 30.01 last 144.94) - open/close 5242 (time 1778535429.97 total 40.02 last 128.53) - open/close 6649 (time 1778535439.97 total 50.02 last 140.65) - open/close 7884 (time 1778535449.97 total 60.02 last 123.49) - open/close 9271 (time 1778535459.98 total 70.02 last 138.70) total: 10000 open/close in 75.19 seconds: 132.99 ops/second - unlinked 0 (time 1778535467 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1573 (time 1778535516.95 total 10.01 last 157.22) - open/close 3158 (time 1778535526.95 total 20.01 last 158.44) - open/close 4608 (time 1778535536.96 total 30.01 last 144.99) - open/close 6117 (time 1778535546.96 total 40.01 last 150.82) - open/close 7468 (time 1778535556.96 total 50.02 last 135.07) - open/close 9165 (time 1778535566.97 total 60.02 last 169.62) total: 10000 open/close in 65.83 seconds: 151.91 ops/second - unlinked 0 (time 1778535574 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1542 (time 1778535625.27 total 10.01 last 154.08) - open/close 3128 (time 1778535635.27 total 20.01 last 158.58) - open/close 4702 (time 1778535645.27 total 30.01 last 157.33) - open/close 6161 (time 1778535655.27 total 40.02 last 145.87) - open/close 7704 (time 1778535665.28 total 50.02 last 154.26) - open/close 9112 (time 1778535675.28 total 60.02 last 140.74) total: 10000 open/close in 64.74 seconds: 154.47 ops/second - unlinked 0 (time 1778535681 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1427 (time 1778535728.66 total 10.00 last 142.63) - open/close 2977 (time 1778535738.66 total 20.01 last 154.94) - open/close 4651 (time 1778535748.67 total 30.02 last 167.28) - open/close 6129 (time 1778535758.67 total 40.02 last 147.77) - open/close 7725 (time 1778535768.67 total 50.02 last 159.57) - open/close 9675 (time 1778535778.67 total 60.02 last 194.97) total: 10000 open/close in 61.47 seconds: 162.68 ops/second - unlinked 0 (time 1778535781 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 1801 (time 1778535818.96 total 10.01 last 180.01) - open/close 3413 (time 1778535828.97 total 20.01 last 161.16) - open/close 5356 (time 1778535838.97 total 30.01 last 194.25) - open/close 7393 (time 1778535848.97 total 40.01 last 203.65) - open/close 9516 (time 1778535858.98 total 50.02 last 212.19) total: 10000 open/close in 52.87 seconds: 189.14 ops/second - unlinked 0 (time 1778535863 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 1870 (time 1778535904.37 total 10.00 last 186.98) - open/close 4056 (time 1778535914.38 total 20.01 last 218.42) - open/close 6256 (time 1778535924.38 total 30.02 last 219.85) - open/close 7925 (time 1778535934.39 total 40.02 last 166.83) - open/close 9791 (time 1778535944.39 total 50.02 last 186.55) total: 10000 open/close in 51.20 seconds: 195.32 ops/second - unlinked 0 (time 1778535947 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 1697 (time 1778535992.05 total 10.01 last 169.61) - open/close 3367 (time 1778536002.06 total 20.02 last 166.82) - open/close 4975 (time 1778536012.07 total 30.02 last 160.69) - open/close 6753 (time 1778536022.07 total 40.02 last 177.79) - open/close 8514 (time 1778536032.07 total 50.02 last 176.09) total: 10000 open/close in 57.19 seconds: 174.84 ops/second - unlinked 0 (time 1778536040 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 1780 (time 1778536085.67 total 10.01 last 177.85) - open/close 3104 (time 1778536095.68 total 20.02 last 132.30) - open/close 4388 (time 1778536105.69 total 30.03 last 128.27) - open/close 5792 (time 1778536115.69 total 40.03 last 140.37) - open/close 7218 (time 1778536125.70 total 50.04 last 142.49) - open/close 8584 (time 1778536135.70 total 60.04 last 136.60) - open/close 9903 (time 1778536145.70 total 70.04 last 131.84) total: 10000 open/close in 70.85 seconds: 141.15 ops/second - unlinked 0 (time 1778536148 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 4096 377472 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272669 s, 3.8 MB/s fail_loc=0x80000204 Failing ost1 on oleg459-server Stopping /mnt/lustre-ost1 (opts:) on oleg459-server reboot facets: ost1 Failover ost1 to oleg459-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-OST0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1778536227.50 total 3.64 last 0.00) total: 100 open/close in 4.81 seconds: 20.80 ops/second umount lustre on /mnt/lustre..... Stopping client oleg459-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing unload_modules_local oleg459-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg459-server: LNET ready to unload oleg459-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.