== conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 16:43:03 (1771969383) Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 oleg255-server: oleg255-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg255-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg255-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 49667 inodes available. Want 100500. rc=0 - open/close 1771 (time 1771969473.87 total 10.00 last 177.02) - open/close 3323 (time 1771969483.87 total 20.01 last 155.14) - open/close 4766 (time 1771969493.88 total 30.02 last 144.17) - open/close 6408 (time 1771969503.88 total 40.02 last 164.18) - open/close 7843 (time 1771969513.89 total 50.03 last 143.38) - open/close 9609 (time 1771969523.90 total 60.03 last 176.53) total: 10000 open/close in 62.27 seconds: 160.59 ops/second - unlinked 0 (time 1771969528 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 1663 (time 1771969573.04 total 10.01 last 166.16) - open/close 3173 (time 1771969583.04 total 20.01 last 151.00) - open/close 4778 (time 1771969593.06 total 30.02 last 160.32) - open/close 6685 (time 1771969603.06 total 40.02 last 190.64) - open/close 8639 (time 1771969613.06 total 50.03 last 195.36) total: 10000 open/close in 58.14 seconds: 172.00 ops/second - unlinked 0 (time 1771969623 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1970 (time 1771969666.74 total 10.00 last 196.99) - open/close 4068 (time 1771969676.74 total 20.01 last 209.70) - open/close 5947 (time 1771969686.74 total 30.01 last 187.87) - open/close 8012 (time 1771969696.75 total 40.02 last 206.33) total: 10000 open/close in 49.91 seconds: 200.35 ops/second - unlinked 0 (time 1771969708 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 1892 (time 1771969744.11 total 10.01 last 189.05) - open/close 3565 (time 1771969754.11 total 20.01 last 167.27) - open/close 5869 (time 1771969764.11 total 30.01 last 230.40) - open/close 7884 (time 1771969774.11 total 40.01 last 201.41) total: 10000 open/close in 48.37 seconds: 206.73 ops/second - unlinked 0 (time 1771969784 ; total 0 ; last 0) total: 10000 unlinks in 28 seconds: 357.142853 unlinks/second - open/close 1930 (time 1771969823.80 total 10.00 last 192.92) - open/close 3875 (time 1771969833.81 total 20.01 last 194.41) - open/close 5710 (time 1771969843.81 total 30.01 last 183.48) - open/close 7500 (time 1771969853.81 total 40.01 last 178.97) - open/close 9211 (time 1771969863.81 total 50.02 last 170.99) total: 10000 open/close in 54.17 seconds: 184.61 ops/second - unlinked 0 (time 1771969869 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1895 (time 1771969910.97 total 10.00 last 189.45) - open/close 4021 (time 1771969920.97 total 20.00 last 212.58) - open/close 6192 (time 1771969930.97 total 30.00 last 217.07) - open/close 8109 (time 1771969940.98 total 40.01 last 191.66) - open/close 9788 (time 1771969950.98 total 50.01 last 167.89) total: 10000 open/close in 51.09 seconds: 195.75 ops/second - unlinked 0 (time 1771969953 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2170 (time 1771969989.08 total 10.01 last 216.87) - open/close 4328 (time 1771969999.08 total 20.01 last 215.72) - open/close 6755 (time 1771970009.08 total 30.01 last 242.66) - open/close 9100 (time 1771970019.08 total 40.01 last 234.50) total: 10000 open/close in 43.94 seconds: 227.60 ops/second - unlinked 0 (time 1771970024 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2383 (time 1771970060.23 total 10.00 last 238.20) - open/close 4528 (time 1771970070.23 total 20.01 last 214.44) - open/close 7190 (time 1771970080.23 total 30.01 last 266.19) total: 10000 open/close in 39.34 seconds: 254.22 ops/second - unlinked 0 (time 1771970091 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 2182 (time 1771970125.38 total 10.00 last 218.10) - open/close 4729 (time 1771970135.38 total 20.01 last 254.66) - open/close 7189 (time 1771970145.39 total 30.01 last 245.85) total: 10000 open/close in 39.94 seconds: 250.38 ops/second - unlinked 0 (time 1771970156 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2218 (time 1771970192.24 total 10.00 last 221.79) - open/close 4454 (time 1771970202.24 total 20.00 last 223.52) - open/close 7111 (time 1771970212.24 total 30.01 last 265.58) - open/close 9420 (time 1771970222.24 total 40.01 last 230.90) total: 10000 open/close in 42.02 seconds: 237.99 ops/second - unlinked 0 (time 1771970225 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 1743 (time 1771970263.21 total 10.00 last 174.23) - open/close 3777 (time 1771970273.21 total 20.01 last 203.35) - open/close 5760 (time 1771970283.21 total 30.01 last 198.27) - open/close 7830 (time 1771970293.22 total 40.01 last 206.95) - open/close 9829 (time 1771970303.22 total 50.01 last 199.87) total: 10000 open/close in 50.72 seconds: 197.17 ops/second - unlinked 0 (time 1771970305 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 4192 82400 5% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 95248 1504 85088 2% /mnt/lustre[MDT:1] lustre-OST0000_UUID 142216 1644 126572 2% /mnt/lustre[OST:0] filesystem_summary: 142216 1644 126572 2% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102823 s, 10.2 MB/s fail_loc=0x80000204 Failing ost1 on oleg255-server Stopping /mnt/lustre-ost1 (opts:) on oleg255-server reboot facets: ost1 Failover ost1 to oleg255-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec total: 100 open/close in 0.74 seconds: 134.30 ops/second umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local oleg255-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg255-server: LNET ready to unload oleg255-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded.