== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:33:46 (1734258826) Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 oleg339-server: oleg339-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: detected 4 online CPUs by sysfs oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg339-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg339-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 302 32594 1% /mnt/lustre[OST:0] filesystem_summary: 32866 272 32594 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1793 (time 1734258880.62 total 10.00 last 179.22) - open/close 3648 (time 1734258890.62 total 20.01 last 185.41) - open/close 5428 (time 1734258900.62 total 30.01 last 177.97) - open/close 7144 (time 1734258910.63 total 40.02 last 171.51) - open/close 8809 (time 1734258920.63 total 50.02 last 166.46) - open/close 10000 (time 1734258927.62 total 57.00 last 170.51) - open/close 11673 (time 1734258937.62 total 67.01 last 167.22) - open/close 13480 (time 1734258947.62 total 77.01 last 180.67) - open/close 15236 (time 1734258957.63 total 87.01 last 175.52) - open/close 17102 (time 1734258967.63 total 97.02 last 186.52) - open/close 18862 (time 1734258977.64 total 107.02 last 175.91) - open/close 20000 (time 1734258983.93 total 113.32 last 180.82) - open/close 22016 (time 1734258993.93 total 123.32 last 201.56) - open/close 24061 (time 1734259003.94 total 133.32 last 204.43) - open/close 26267 (time 1734259013.94 total 143.32 last 220.57) - open/close 28175 (time 1734259023.94 total 153.33 last 190.74) - open/close 30000 (time 1734259033.91 total 163.29 last 183.10) - open/close 31763 (time 1734259043.91 total 173.30 last 176.20) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 178.37 seconds: 182.92 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3500 (time 1734259060.38 total 10.00 last 349.95) - /unlink 6622 (time 1734259070.39 total 20.01 last 312.04) - /unlink 9617 (time 1734259080.39 total 30.01 last 299.45) - /unlink 12815 (time 1734259090.39 total 40.01 last 319.77) - /unlink 15911 (time 1734259100.39 total 50.01 last 309.58) - /unlink 19046 (time 1734259110.39 total 60.01 last 313.48) - /unlink 20000 (time 1734259112.91 total 62.53 last 378.46) - /unlink 24022 (time 1734259122.91 total 72.53 last 402.16) - /unlink 27425 (time 1734259132.92 total 82.53 last 340.27) - /unlink 30000 (time 1734259140.81 total 90.43 last 325.97) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 98.22 seconds: 332.17 ops/second Waiting for MDT destroys to complete oleg339-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg339-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 - open/close 2009 (time 1734259169.52 total 10.00 last 200.83) - open/close 4060 (time 1734259179.52 total 20.01 last 205.05) - open/close 5933 (time 1734259189.52 total 30.01 last 187.23) - open/close 7892 (time 1734259199.52 total 40.01 last 195.88) - open/close 9765 (time 1734259209.53 total 50.01 last 187.28) - open/close 11587 (time 1734259219.53 total 60.01 last 182.17) - open/close 13541 (time 1734259229.53 total 70.02 last 195.34) - open/close 15502 (time 1734259239.54 total 80.02 last 196.01) - open/close 17948 (time 1734259249.54 total 90.02 last 244.55) - open/close 20000 (time 1734259257.76 total 98.25 last 249.44) - open/close 22165 (time 1734259267.77 total 108.25 last 216.42) - open/close 24018 (time 1734259277.77 total 118.26 last 185.24) - open/close 26202 (time 1734259287.77 total 128.26 last 218.32) - open/close 28147 (time 1734259297.78 total 138.27 last 194.37) - open/close 30000 (time 1734259306.93 total 147.42 last 202.54) - open/close 31930 (time 1734259316.93 total 157.42 last 192.95) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 161.03 seconds: 202.61 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4740 (time 1734259331.97 total 10.00 last 473.93) - /unlink 8982 (time 1734259341.97 total 20.00 last 424.15) - /unlink 10000 (time 1734259344.52 total 22.55 last 400.06) - /unlink 13513 (time 1734259354.52 total 32.55 last 351.23) - /unlink 16996 (time 1734259364.52 total 42.55 last 348.25) - /unlink 20000 (time 1734259372.59 total 50.62 last 372.19) - /unlink 23711 (time 1734259382.59 total 60.62 last 371.06) - /unlink 27343 (time 1734259392.59 total 70.62 last 363.16) - /unlink 30000 (time 1734259399.40 total 77.43 last 390.11) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 84.34 seconds: 386.84 ops/second Waiting for MDT destroys to complete oleg339-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 oleg339-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 - open/close 1945 (time 1734259424.64 total 10.00 last 194.50) - open/close 3891 (time 1734259434.64 total 20.00 last 194.54) - open/close 5896 (time 1734259444.64 total 30.01 last 200.39) - open/close 7915 (time 1734259454.65 total 40.01 last 201.83) - open/close 9801 (time 1734259464.65 total 50.02 last 188.53) - open/close 11797 (time 1734259474.65 total 60.02 last 199.57) - open/close 13962 (time 1734259484.65 total 70.02 last 216.48) - open/close 16142 (time 1734259494.65 total 80.02 last 218.00) - open/close 18068 (time 1734259504.66 total 90.02 last 192.53) - open/close 20000 (time 1734259514.32 total 99.69 last 199.85) - open/close 22221 (time 1734259524.33 total 109.69 last 222.07) - open/close 24825 (time 1734259534.33 total 119.69 last 260.29) - open/close 26980 (time 1734259544.33 total 129.70 last 215.45) - open/close 29080 (time 1734259554.33 total 139.70 last 209.98) - open/close 30000 (time 1734259558.17 total 143.54 last 239.55) - open/close 32116 (time 1734259568.18 total 153.54 last 211.56) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 156.40 seconds: 208.60 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4064 (time 1734259582.38 total 10.00 last 406.36) - /unlink 8479 (time 1734259592.39 total 20.01 last 441.31) - /unlink 10000 (time 1734259595.85 total 23.47 last 439.37) - /unlink 15324 (time 1734259605.85 total 33.47 last 532.35) - /unlink 20000 (time 1734259614.40 total 42.02 last 546.77) - /unlink 25210 (time 1734259624.40 total 52.02 last 520.98) - /unlink 30000 (time 1734259633.96 total 61.57 last 501.35) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 66.83 seconds: 488.22 ops/second Waiting for MDT destroys to complete oleg339-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg339-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 - open/close 2689 (time 1734259656.41 total 10.00 last 268.79) - open/close 5165 (time 1734259666.41 total 20.00 last 247.59) - open/close 7793 (time 1734259676.41 total 30.01 last 262.73) - open/close 10000 (time 1734259684.83 total 38.42 last 262.25) - open/close 12614 (time 1734259694.83 total 48.42 last 261.38) - open/close 15594 (time 1734259704.83 total 58.43 last 297.94) - open/close 18305 (time 1734259714.83 total 68.43 last 271.01) - open/close 20000 (time 1734259721.77 total 75.37 last 244.17) - open/close 22339 (time 1734259731.77 total 85.37 last 233.89) - open/close 24498 (time 1734259741.78 total 95.37 last 215.85) - open/close 26674 (time 1734259751.78 total 105.38 last 217.57) - open/close 28821 (time 1734259761.78 total 115.38 last 214.69) - open/close 30000 (time 1734259766.39 total 119.99 last 255.36) - open/close 32540 (time 1734259776.40 total 129.99 last 253.94) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 130.28 seconds: 250.43 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 5132 (time 1734259787.76 total 10.00 last 513.18) - /unlink 9956 (time 1734259797.76 total 20.00 last 482.33) - /unlink 13849 (time 1734259807.77 total 30.00 last 389.19) - /unlink 18580 (time 1734259817.77 total 40.01 last 473.03) - /unlink 20000 (time 1734259820.62 total 42.86 last 497.91) - /unlink 25699 (time 1734259830.62 total 52.86 last 569.87) - /unlink 30000 (time 1734259838.42 total 60.66 last 551.23) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 65.26 seconds: 499.91 ops/second Waiting for MDT destroys to complete oleg339-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg339-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 - open/close 2937 (time 1734259864.41 total 10.00 last 293.70) - open/close 5405 (time 1734259874.41 total 20.00 last 246.80) - open/close 7792 (time 1734259884.41 total 30.00 last 238.66) - open/close 10000 (time 1734259894.19 total 39.78 last 225.74) - open/close 12251 (time 1734259904.20 total 49.79 last 225.04) - open/close 14766 (time 1734259914.20 total 59.79 last 251.46) - open/close 17548 (time 1734259924.20 total 69.79 last 278.17) - open/close 20000 (time 1734259933.55 total 79.14 last 262.13) - open/close 22590 (time 1734259943.55 total 89.14 last 258.98) - open/close 25141 (time 1734259953.56 total 99.15 last 255.02) - open/close 27626 (time 1734259963.56 total 109.15 last 248.41) - open/close 30000 (time 1734259972.57 total 118.16 last 263.60) - open/close 32527 (time 1734259982.57 total 128.16 last 252.63) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 128.52 seconds: 253.85 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 5382 (time 1734259994.01 total 10.01 last 537.56) - /unlink 10000 (time 1734260001.55 total 17.54 last 613.14) - /unlink 16644 (time 1734260011.55 total 27.54 last 664.38) - /unlink 20000 (time 1734260017.34 total 33.33 last 579.66) - /unlink 25633 (time 1734260027.34 total 43.33 last 563.27) - /unlink 30000 (time 1734260034.23 total 50.23 last 633.45) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 54.90 seconds: 594.24 ops/second Waiting for MDT destroys to complete oleg339-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 oleg339-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2