== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 05:41:05 (1774518065) Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f) oleg237-server: oleg237-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg237-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg237-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg237-client.virtnet: -o user_xattr,flock oleg237-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 11872 inodes available. Want 100500. rc=0 - open/close 649 (time 1774518221.80 total 10.01 last 64.87) - open/close 1287 (time 1774518231.80 total 20.01 last 63.78) - open/close 1923 (time 1774518241.81 total 30.02 last 63.55) - open/close 2617 (time 1774518251.82 total 40.02 last 69.38) - open/close 3313 (time 1774518261.82 total 50.02 last 69.57) - open/close 4041 (time 1774518271.83 total 60.03 last 72.73) - open/close 4702 (time 1774518281.84 total 70.04 last 66.06) - open/close 5423 (time 1774518291.84 total 80.04 last 72.07) - open/close 5955 (time 1774518301.85 total 90.05 last 53.14) - open/close 6452 (time 1774518311.86 total 100.07 last 49.64) - open/close 6953 (time 1774518321.87 total 110.07 last 50.06) - open/close 7603 (time 1774518331.87 total 120.08 last 64.97) - open/close 8356 (time 1774518341.88 total 130.09 last 75.23) - open/close 9064 (time 1774518351.90 total 140.10 last 70.69) - open/close 9820 (time 1774518361.90 total 150.11 last 75.59) total: 10000 open/close in 152.32 seconds: 65.65 ops/second - unlinked 0 (time 1774518369 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 672 (time 1774518482.65 total 10.01 last 67.16) - open/close 1382 (time 1774518492.66 total 20.01 last 70.97) - open/close 2081 (time 1774518502.66 total 30.02 last 69.86) - open/close 2779 (time 1774518512.69 total 40.04 last 69.62) - open/close 3476 (time 1774518522.70 total 50.05 last 69.61) - open/close 4156 (time 1774518532.70 total 60.06 last 67.98) - open/close 4854 (time 1774518542.72 total 70.08 last 69.67) - open/close 5576 (time 1774518552.73 total 80.08 last 72.13) - open/close 6354 (time 1774518562.75 total 90.10 last 77.67) - open/close 7093 (time 1774518572.76 total 100.11 last 73.82) - open/close 7837 (time 1774518582.76 total 110.11 last 74.39) - open/close 8623 (time 1774518592.76 total 120.12 last 78.56) - open/close 9468 (time 1774518602.77 total 130.12 last 84.45) total: 10000 open/close in 136.42 seconds: 73.30 ops/second - unlinked 0 (time 1774518611 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 920 (time 1774518692.38 total 10.01 last 91.92) - open/close 1689 (time 1774518702.39 total 20.02 last 76.79) - open/close 2607 (time 1774518712.40 total 30.03 last 91.72) - open/close 3419 (time 1774518722.41 total 40.04 last 81.16) - open/close 4357 (time 1774518732.42 total 50.05 last 93.72) - open/close 5502 (time 1774518742.42 total 60.05 last 114.43) - open/close 6569 (time 1774518752.43 total 70.06 last 106.60) - open/close 7535 (time 1774518762.44 total 80.07 last 96.51) - open/close 8442 (time 1774518772.44 total 90.07 last 90.69) - open/close 9533 (time 1774518782.45 total 100.08 last 109.04) total: 10000 open/close in 103.21 seconds: 96.89 ops/second - unlinked 0 (time 1774518787 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1701 (time 1774518847.59 total 10.01 last 170.00) - open/close 3394 (time 1774518857.59 total 20.01 last 169.29) - open/close 4560 (time 1774518867.61 total 30.02 last 116.41) - open/close 5655 (time 1774518877.61 total 40.02 last 109.48) - open/close 6684 (time 1774518887.61 total 50.03 last 102.87) - open/close 7915 (time 1774518897.62 total 60.03 last 123.03) - open/close 9090 (time 1774518907.62 total 70.04 last 117.45) total: 10000 open/close in 77.69 seconds: 128.72 ops/second - unlinked 0 (time 1774518917 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1138 (time 1774518972.58 total 10.01 last 113.69) - open/close 2254 (time 1774518982.59 total 20.01 last 111.56) - open/close 3536 (time 1774518992.59 total 30.02 last 128.12) - open/close 4674 (time 1774519002.59 total 40.02 last 113.77) - open/close 5902 (time 1774519012.60 total 50.02 last 122.77) - open/close 7116 (time 1774519022.60 total 60.03 last 121.38) - open/close 8416 (time 1774519032.60 total 70.03 last 129.96) - open/close 9605 (time 1774519042.61 total 80.03 last 118.84) total: 10000 open/close in 82.96 seconds: 120.54 ops/second - unlinked 0 (time 1774519047 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1228 (time 1774519106.02 total 10.01 last 122.71) - open/close 2472 (time 1774519116.03 total 20.01 last 124.33) - open/close 3736 (time 1774519126.03 total 30.01 last 126.40) - open/close 4932 (time 1774519136.03 total 40.02 last 119.52) - open/close 5991 (time 1774519146.04 total 50.03 last 105.83) - open/close 7233 (time 1774519156.05 total 60.03 last 124.11) - open/close 8430 (time 1774519166.06 total 70.05 last 119.50) - open/close 9608 (time 1774519176.07 total 80.06 last 117.72) total: 10000 open/close in 83.43 seconds: 119.86 ops/second - unlinked 0 (time 1774519182 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1302 (time 1774519248.23 total 10.00 last 130.18) - open/close 2562 (time 1774519258.23 total 20.00 last 125.98) - open/close 3847 (time 1774519268.24 total 30.00 last 128.48) - open/close 5201 (time 1774519278.24 total 40.01 last 135.28) - open/close 6415 (time 1774519288.25 total 50.02 last 121.35) - open/close 7848 (time 1774519298.26 total 60.03 last 143.12) - open/close 9319 (time 1774519308.27 total 70.04 last 146.97) total: 10000 open/close in 75.30 seconds: 132.79 ops/second - unlinked 0 (time 1774519315 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1178 (time 1774519372.65 total 10.00 last 117.77) - open/close 2284 (time 1774519382.65 total 20.01 last 110.55) - open/close 3654 (time 1774519392.66 total 30.02 last 136.88) - open/close 4942 (time 1774519402.66 total 40.02 last 128.77) - open/close 6433 (time 1774519412.67 total 50.02 last 149.02) - open/close 8024 (time 1774519422.67 total 60.03 last 159.03) - open/close 9670 (time 1774519432.68 total 70.03 last 164.58) total: 10000 open/close in 73.00 seconds: 136.98 ops/second - unlinked 0 (time 1774519438 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1631 (time 1774519486.73 total 10.00 last 163.04) - open/close 2953 (time 1774519496.74 total 20.00 last 132.19) - open/close 4416 (time 1774519506.74 total 30.00 last 146.29) - open/close 5770 (time 1774519516.74 total 40.01 last 135.40) - open/close 7089 (time 1774519526.74 total 50.01 last 131.83) - open/close 8297 (time 1774519536.74 total 60.01 last 120.78) - open/close 9497 (time 1774519546.75 total 70.01 last 119.97) total: 10000 open/close in 74.08 seconds: 134.99 ops/second - unlinked 0 (time 1774519553 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1349 (time 1774519605.68 total 10.00 last 134.89) - open/close 2857 (time 1774519615.68 total 20.01 last 150.72) - open/close 4265 (time 1774519625.69 total 30.01 last 140.73) - open/close 5657 (time 1774519635.69 total 40.02 last 139.15) - open/close 6915 (time 1774519645.69 total 50.02 last 125.79) - open/close 8290 (time 1774519655.69 total 60.02 last 137.50) - open/close 9592 (time 1774519665.70 total 70.02 last 130.16) total: 10000 open/close in 73.47 seconds: 136.11 ops/second - unlinked 0 (time 1774519671 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1276 (time 1774519727.05 total 10.01 last 127.53) - open/close 2536 (time 1774519737.06 total 20.02 last 125.87) - open/close 3776 (time 1774519747.07 total 30.02 last 123.95) - open/close 5095 (time 1774519757.07 total 40.02 last 131.84) - open/close 6317 (time 1774519767.08 total 50.03 last 122.17) - open/close 7384 (time 1774519777.08 total 60.03 last 106.69) - open/close 8715 (time 1774519787.08 total 70.03 last 133.05) total: 10000 open/close in 78.98 seconds: 126.61 ops/second - unlinked 0 (time 1774519798 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 5504 376064 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 3072 377856 1% /mnt/lustre[OST:0] filesystem_summary: 382976 3072 377856 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188006 s, 5.6 MB/s fail_loc=0x80000204 Failing ost1 on oleg237-server Stopping /mnt/lustre-ost1 (opts:) on oleg237-server reboot facets: ost1 Failover ost1 to oleg237-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 100 open/close in 1.03 seconds: 96.82 ops/second umount lustre on /mnt/lustre..... Stopping client oleg237-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg237-server: LNET ready to unload oleg237-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.