== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 16:33:28 (1771968808) Stopping clients: oleg230-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg230-client.virtnet /mnt/lustre2 (opts:-f) oleg230-server: oleg230-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg230-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg230-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg230-client.virtnet: -o user_xattr,flock oleg230-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 973 (time 1771968900.10 total 10.00 last 97.28) - open/close 1991 (time 1771968910.11 total 20.01 last 101.73) - open/close 3223 (time 1771968920.11 total 30.01 last 123.17) - open/close 4318 (time 1771968930.11 total 40.02 last 109.44) - open/close 5558 (time 1771968940.12 total 50.02 last 123.97) - open/close 6730 (time 1771968950.12 total 60.02 last 117.17) - open/close 7891 (time 1771968960.12 total 70.03 last 116.06) - open/close 9021 (time 1771968970.13 total 80.03 last 112.94) total: 10000 open/close in 89.35 seconds: 111.91 ops/second - unlinked 0 (time 1771968981 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1084 (time 1771969045.42 total 10.00 last 108.39) - open/close 2171 (time 1771969055.42 total 20.00 last 108.69) - open/close 3227 (time 1771969065.42 total 30.00 last 105.57) - open/close 4317 (time 1771969075.43 total 40.01 last 108.89) - open/close 5463 (time 1771969085.45 total 50.03 last 114.41) - open/close 6478 (time 1771969095.45 total 60.03 last 101.49) - open/close 7564 (time 1771969105.46 total 70.04 last 108.53) - open/close 8668 (time 1771969115.46 total 80.04 last 110.38) - open/close 9733 (time 1771969125.47 total 90.05 last 106.36) total: 10000 open/close in 92.94 seconds: 107.60 ops/second - unlinked 0 (time 1771969130 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1289 (time 1771969183.32 total 10.00 last 128.85) - open/close 2556 (time 1771969193.32 total 20.01 last 126.65) - open/close 3723 (time 1771969203.34 total 30.02 last 116.54) - open/close 4876 (time 1771969213.34 total 40.03 last 115.26) - open/close 6122 (time 1771969223.35 total 50.04 last 124.43) - open/close 7276 (time 1771969233.35 total 60.04 last 115.39) - open/close 8554 (time 1771969243.36 total 70.04 last 127.77) - open/close 9798 (time 1771969253.36 total 80.05 last 124.34) total: 10000 open/close in 81.76 seconds: 122.31 ops/second - unlinked 0 (time 1771969257 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1140 (time 1771969310.52 total 10.00 last 113.97) - open/close 2415 (time 1771969320.52 total 20.01 last 127.46) - open/close 3666 (time 1771969330.53 total 30.01 last 125.01) - open/close 4888 (time 1771969340.53 total 40.02 last 122.15) - open/close 6248 (time 1771969350.54 total 50.02 last 135.90) - open/close 7580 (time 1771969360.55 total 60.03 last 133.08) - open/close 9156 (time 1771969370.55 total 70.03 last 157.60) total: 10000 open/close in 76.42 seconds: 130.85 ops/second - unlinked 0 (time 1771969378 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1874 (time 1771969428.89 total 10.01 last 187.28) - open/close 3345 (time 1771969438.89 total 20.01 last 147.08) - open/close 4751 (time 1771969448.89 total 30.01 last 140.59) - open/close 6235 (time 1771969458.89 total 40.01 last 148.36) - open/close 7857 (time 1771969468.89 total 50.01 last 162.19) - open/close 9566 (time 1771969478.90 total 60.02 last 170.83) total: 10000 open/close in 62.55 seconds: 159.86 ops/second - unlinked 0 (time 1771969483 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1600 (time 1771969531.25 total 10.00 last 159.97) - open/close 2920 (time 1771969541.26 total 20.01 last 131.86) - open/close 4369 (time 1771969551.27 total 30.03 last 144.72) - open/close 6040 (time 1771969561.27 total 40.03 last 167.09) - open/close 7539 (time 1771969571.28 total 50.03 last 149.87) - open/close 8960 (time 1771969581.28 total 60.03 last 142.06) total: 10000 open/close in 66.70 seconds: 149.93 ops/second - unlinked 0 (time 1771969590 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1359 (time 1771969633.71 total 10.00 last 135.87) - open/close 3075 (time 1771969643.71 total 20.01 last 171.54) - open/close 4653 (time 1771969653.71 total 30.01 last 157.80) - open/close 6231 (time 1771969663.71 total 40.01 last 157.79) - open/close 8223 (time 1771969673.71 total 50.01 last 199.11) total: 10000 open/close in 58.91 seconds: 169.76 ops/second - unlinked 0 (time 1771969684 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second - open/close 1955 (time 1771969723.45 total 10.01 last 195.37) - open/close 3714 (time 1771969733.46 total 20.01 last 175.85) - open/close 5442 (time 1771969743.46 total 30.01 last 172.78) - open/close 6990 (time 1771969753.46 total 40.02 last 154.69) - open/close 9154 (time 1771969763.47 total 50.02 last 216.36) total: 10000 open/close in 54.01 seconds: 185.14 ops/second - unlinked 0 (time 1771969769 ; total 0 ; last 0) total: 10000 unlinks in 28 seconds: 357.142853 unlinks/second - open/close 1909 (time 1771969809.22 total 10.01 last 190.80) - open/close 3808 (time 1771969819.22 total 20.01 last 189.83) - open/close 5589 (time 1771969829.23 total 30.01 last 178.05) - open/close 7427 (time 1771969839.23 total 40.02 last 183.73) - open/close 9132 (time 1771969849.24 total 50.02 last 170.41) total: 10000 open/close in 55.58 seconds: 179.93 ops/second - unlinked 0 (time 1771969856 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1812 (time 1771969899.43 total 10.00 last 181.14) - open/close 3563 (time 1771969909.44 total 20.01 last 174.95) - open/close 5523 (time 1771969919.45 total 30.01 last 195.94) - open/close 7714 (time 1771969929.45 total 40.02 last 219.00) - open/close 9568 (time 1771969939.45 total 50.02 last 185.38) total: 10000 open/close in 52.67 seconds: 189.86 ops/second - unlinked 0 (time 1771969943 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 2113 (time 1771969984.47 total 10.01 last 211.19) - open/close 4176 (time 1771969994.47 total 20.01 last 206.15) - open/close 6289 (time 1771970004.48 total 30.01 last 211.27) - open/close 8470 (time 1771970014.48 total 40.01 last 218.08) total: 10000 open/close in 46.45 seconds: 215.27 ops/second - unlinked 0 (time 1771970022 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 5504 376064 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 4096 376832 2% /mnt/lustre[OST:0] filesystem_summary: 382976 4096 376832 2% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281002 s, 3.7 MB/s fail_loc=0x80000204 Failing ost1 on oleg230-server Stopping /mnt/lustre-ost1 (opts:) on oleg230-server reboot facets: ost1 Failover ost1 to oleg230-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1771970075.35 total 3.25 last 0.00) total: 100 open/close in 4.14 seconds: 24.18 ops/second umount lustre on /mnt/lustre..... Stopping client oleg230-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg230-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg230-server stop mds service on oleg230-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg230-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing unload_modules_local oleg230-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg230-server: LNET ready to unload oleg230-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.