== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 04:36:09 (1779698169) Stopping clients: oleg132-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg132-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg132-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg132-server oleg132-server: oleg132-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg132-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg132-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg132-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 657 (time 1779698391.06 total 10.01 last 65.62) - open/close 1470 (time 1779698401.06 total 20.01 last 81.29) - open/close 2119 (time 1779698411.07 total 30.02 last 64.86) - open/close 2883 (time 1779698421.07 total 40.02 last 76.40) - open/close 3691 (time 1779698431.08 total 50.03 last 80.71) - open/close 4310 (time 1779698441.09 total 60.05 last 61.80) total: 4500 open/close in 62.81 seconds: 71.65 ops/second - /unlink 1375 (time 1779698458.85 total 10.00 last 137.49) - /unlink 2514 (time 1779698468.85 total 20.00 last 113.86) - /unlink 3655 (time 1779698478.85 total 30.01 last 114.09) total: 4500 /unlink in 37.69 seconds: 119.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 534 (time 1779698502.11 total 10.01 last 53.34) - open/close 1032 (time 1779698512.14 total 20.04 last 49.65) - open/close 1562 (time 1779698522.15 total 30.04 last 52.99) - open/close 2099 (time 1779698532.15 total 40.05 last 53.68) - open/close 2690 (time 1779698542.16 total 50.06 last 59.04) - open/close 3428 (time 1779698552.16 total 60.06 last 73.79) - open/close 3992 (time 1779698562.17 total 70.06 last 56.37) total: 4500 open/close in 78.19 seconds: 57.55 ops/second - /unlink 1240 (time 1779698586.34 total 10.00 last 123.95) - /unlink 2515 (time 1779698596.34 total 20.01 last 127.42) - /unlink 3765 (time 1779698606.35 total 30.01 last 124.97) total: 4500 /unlink in 36.49 seconds: 123.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 622 (time 1779698629.28 total 10.02 last 62.10) - open/close 1191 (time 1779698639.29 total 20.02 last 56.88) - open/close 1787 (time 1779698649.31 total 30.05 last 59.45) - open/close 2419 (time 1779698659.33 total 40.07 last 63.07) - open/close 2951 (time 1779698669.34 total 50.07 last 53.18) - open/close 3519 (time 1779698679.34 total 60.08 last 56.76) - open/close 4113 (time 1779698689.35 total 70.09 last 59.35) total: 4500 open/close in 76.51 seconds: 58.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1249 (time 1779698711.53 total 10.00 last 124.88) - /unlink 2341 (time 1779698721.54 total 20.00 last 109.17) - /unlink 3564 (time 1779698731.54 total 30.01 last 122.29) total: 4500 /unlink in 37.45 seconds: 120.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 674 (time 1779698754.61 total 10.00 last 67.38) - open/close 1420 (time 1779698764.62 total 20.01 last 74.56) - open/close 2146 (time 1779698774.62 total 30.01 last 72.59) - open/close 2814 (time 1779698784.62 total 40.01 last 66.77) - open/close 3534 (time 1779698794.62 total 50.02 last 71.99) - open/close 4216 (time 1779698804.63 total 60.02 last 68.15) total: 4500 open/close in 65.07 seconds: 69.16 ops/second - /unlink 1301 (time 1779698824.94 total 10.01 last 130.03) - /unlink 2629 (time 1779698834.94 total 20.01 last 132.76) - /unlink 3932 (time 1779698844.95 total 30.01 last 130.26) total: 4500 /unlink in 35.51 seconds: 126.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 575 (time 1779698866.77 total 10.02 last 57.36) - open/close 1116 (time 1779698876.78 total 20.03 last 54.06) - open/close 1672 (time 1779698886.78 total 30.04 last 55.58) - open/close 2377 (time 1779698896.78 total 40.04 last 70.48) - open/close 3088 (time 1779698906.79 total 50.05 last 71.04) - open/close 3700 (time 1779698916.82 total 60.07 last 61.04) - open/close 4293 (time 1779698926.82 total 70.08 last 59.28) total: 4500 open/close in 73.32 seconds: 61.38 ops/second - /unlink 1167 (time 1779698945.81 total 10.02 last 116.48) - /unlink 2322 (time 1779698955.82 total 20.03 last 115.40) - /unlink 3566 (time 1779698965.82 total 30.03 last 124.39) total: 4500 /unlink in 37.45 seconds: 120.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 523 (time 1779698990.53 total 10.01 last 52.27) - open/close 1090 (time 1779699000.54 total 20.01 last 56.67) - open/close 1801 (time 1779699010.55 total 30.02 last 71.03) - open/close 2449 (time 1779699020.59 total 40.06 last 64.56) - open/close 3093 (time 1779699030.59 total 50.06 last 64.38) - open/close 3770 (time 1779699040.59 total 60.07 last 67.68) - open/close 4476 (time 1779699050.60 total 70.07 last 70.58) total: 4500 open/close in 70.38 seconds: 63.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1216 (time 1779699066.67 total 10.01 last 121.50) - /unlink 2446 (time 1779699076.67 total 20.02 last 122.91) - /unlink 3790 (time 1779699086.68 total 30.02 last 134.31) total: 4500 /unlink in 34.64 seconds: 129.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 724 (time 1779699106.82 total 10.01 last 72.33) - open/close 1422 (time 1779699116.83 total 20.02 last 69.75) - open/close 2176 (time 1779699126.83 total 30.02 last 75.36) - open/close 2769 (time 1779699136.87 total 40.06 last 59.09) - open/close 3378 (time 1779699146.88 total 50.06 last 60.87) - open/close 4058 (time 1779699156.89 total 60.08 last 67.87) total: 4500 open/close in 66.42 seconds: 67.76 ops/second - /unlink 1252 (time 1779699179.45 total 10.01 last 125.03) - /unlink 2416 (time 1779699189.45 total 20.02 last 116.37) - /unlink 3657 (time 1779699199.45 total 30.02 last 124.05) total: 4500 /unlink in 36.32 seconds: 123.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 622 (time 1779699221.86 total 10.00 last 62.18) - open/close 1316 (time 1779699231.86 total 20.00 last 69.40) - open/close 2008 (time 1779699241.88 total 30.02 last 69.07) - open/close 2775 (time 1779699251.89 total 40.02 last 76.68) - open/close 3388 (time 1779699261.90 total 50.04 last 61.18) - open/close 4002 (time 1779699271.91 total 60.05 last 61.36) total: 4500 open/close in 66.82 seconds: 67.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1032 (time 1779699295.29 total 10.01 last 103.10) - /unlink 2158 (time 1779699305.29 total 20.01 last 112.57) - /unlink 3342 (time 1779699315.30 total 30.02 last 118.30) total: 4500 /unlink in 38.90 seconds: 115.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 652 (time 1779699340.00 total 10.01 last 65.15) - open/close 1141 (time 1779699350.01 total 20.02 last 48.84) - open/close 1836 (time 1779699360.04 total 30.04 last 69.32) - open/close 2522 (time 1779699370.04 total 40.05 last 68.57) - open/close 3006 (time 1779699380.05 total 50.05 last 48.38) - open/close 3426 (time 1779699390.10 total 60.11 last 41.78) - open/close 3771 (time 1779699400.11 total 70.11 last 34.48) - open/close 4072 (time 1779699410.11 total 80.12 last 30.09) total: 4500 open/close in 87.12 seconds: 51.66 ops/second - /unlink 1200 (time 1779699433.38 total 10.00 last 119.95) - /unlink 2440 (time 1779699443.39 total 20.01 last 123.91) - /unlink 3771 (time 1779699453.39 total 30.01 last 133.07) total: 4500 /unlink in 35.73 seconds: 125.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 571 (time 1779699474.27 total 10.01 last 57.04) - open/close 1203 (time 1779699484.28 total 20.01 last 63.18) - open/close 1901 (time 1779699494.28 total 30.02 last 69.75) - open/close 2658 (time 1779699504.29 total 40.02 last 75.68) - open/close 3325 (time 1779699514.29 total 50.03 last 66.66) - open/close 3919 (time 1779699524.31 total 60.04 last 59.33) - open/close 4469 (time 1779699534.32 total 70.05 last 54.94) total: 4500 open/close in 70.51 seconds: 63.82 ops/second - /unlink 1176 (time 1779699551.65 total 10.00 last 117.59) - /unlink 2164 (time 1779699561.65 total 20.00 last 98.77) - /unlink 3410 (time 1779699571.65 total 30.01 last 124.56) - /unlink 4429 (time 1779699581.66 total 40.01 last 101.84) total: 4500 /unlink in 40.86 seconds: 110.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 510 (time 1779699598.28 total 10.03 last 50.87) - open/close 1055 (time 1779699608.30 total 20.04 last 54.40) - open/close 1666 (time 1779699618.30 total 30.05 last 61.09) - open/close 2083 (time 1779699628.31 total 40.05 last 41.67) - open/close 2450 (time 1779699638.31 total 50.05 last 36.70) - open/close 3019 (time 1779699648.31 total 60.06 last 56.89) - open/close 3534 (time 1779699658.34 total 70.09 last 51.33) - open/close 4083 (time 1779699668.36 total 80.10 last 54.81) total: 4500 open/close in 87.63 seconds: 51.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1073 (time 1779699692.50 total 10.01 last 107.22) - /unlink 2255 (time 1779699702.51 total 20.01 last 118.14) - /unlink 3201 (time 1779699712.51 total 30.01 last 94.58) - /unlink 4232 (time 1779699722.51 total 40.02 last 103.06) total: 4500 /unlink in 42.15 seconds: 106.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 617 (time 1779699741.11 total 10.01 last 61.62) - open/close 1148 (time 1779699751.11 total 20.02 last 53.09) - open/close 1744 (time 1779699761.11 total 30.02 last 59.58) - open/close 2333 (time 1779699771.11 total 40.02 last 58.89) - open/close 3031 (time 1779699781.13 total 50.04 last 69.71) - open/close 3653 (time 1779699791.14 total 60.05 last 62.10) - open/close 4309 (time 1779699801.15 total 70.06 last 65.56) total: 4500 open/close in 73.16 seconds: 61.51 ops/second - /unlink 1157 (time 1779699820.16 total 10.02 last 115.44) - /unlink 2155 (time 1779699830.16 total 20.02 last 99.80) - /unlink 3418 (time 1779699840.16 total 30.03 last 126.24) total: 4500 /unlink in 39.25 seconds: 114.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 687 (time 1779699866.15 total 10.00 last 68.68) - open/close 1363 (time 1779699876.15 total 20.01 last 67.57) - open/close 1930 (time 1779699886.15 total 30.01 last 56.69) - open/close 2539 (time 1779699896.16 total 40.01 last 60.87) - open/close 3179 (time 1779699906.17 total 50.02 last 63.94) - open/close 3860 (time 1779699916.18 total 60.04 last 67.99) total: 4500 open/close in 69.92 seconds: 64.36 ops/second - /unlink 1204 (time 1779699942.48 total 10.01 last 120.25) - /unlink 2281 (time 1779699952.50 total 20.03 last 107.48) - /unlink 3539 (time 1779699962.50 total 30.03 last 125.77) total: 4500 /unlink in 36.95 seconds: 121.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 683 (time 1779699985.43 total 10.01 last 68.20) - open/close 1341 (time 1779699995.43 total 20.02 last 65.79) - open/close 1962 (time 1779700005.45 total 30.04 last 61.98) - open/close 2667 (time 1779700015.48 total 40.06 last 70.35) - open/close 3175 (time 1779700025.48 total 50.06 last 50.79) - open/close 3895 (time 1779700035.48 total 60.07 last 71.95) total: 4500 open/close in 68.79 seconds: 65.41 ops/second - /unlink 1236 (time 1779700059.56 total 10.00 last 123.59) - /unlink 2494 (time 1779700069.56 total 20.00 last 125.76) - /unlink 3854 (time 1779700079.57 total 30.01 last 135.95) total: 4500 /unlink in 35.71 seconds: 126.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 598 (time 1779700101.52 total 10.02 last 59.66) - open/close 1312 (time 1779700111.54 total 20.05 last 71.22) - open/close 1998 (time 1779700121.55 total 30.06 last 68.55) - open/close 2572 (time 1779700131.55 total 40.06 last 57.38) - open/close 3123 (time 1779700141.60 total 50.11 last 54.83) - open/close 3692 (time 1779700151.61 total 60.12 last 56.84) - open/close 4152 (time 1779700161.62 total 70.13 last 45.97) total: 4500 open/close in 76.39 seconds: 58.91 ops/second - /unlink 1247 (time 1779700183.86 total 10.01 last 124.63) - /unlink 2489 (time 1779700193.87 total 20.01 last 124.15) - /unlink 3434 (time 1779700203.87 total 30.01 last 94.47) - /unlink 4399 (time 1779700213.87 total 40.01 last 96.48) total: 4500 /unlink in 41.02 seconds: 109.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 639 (time 1779700232.06 total 10.03 last 63.71) - open/close 1206 (time 1779700242.06 total 20.03 last 56.69) - open/close 1882 (time 1779700252.07 total 30.04 last 67.55) - open/close 2594 (time 1779700262.08 total 40.05 last 71.15) - open/close 3261 (time 1779700272.08 total 50.06 last 66.64) - open/close 4021 (time 1779700282.09 total 60.06 last 75.94) total: 4500 open/close in 66.40 seconds: 67.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1129 (time 1779700305.16 total 10.01 last 112.76) - /unlink 2179 (time 1779700315.16 total 20.01 last 104.99) - /unlink 3171 (time 1779700325.17 total 30.02 last 99.12) - /unlink 4179 (time 1779700335.17 total 40.02 last 100.76) total: 4500 /unlink in 42.56 seconds: 105.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 676 (time 1779700354.83 total 10.01 last 67.54) - open/close 1421 (time 1779700364.83 total 20.02 last 74.44) - open/close 2020 (time 1779700374.87 total 30.06 last 59.67) - open/close 2715 (time 1779700384.90 total 40.08 last 69.33) - open/close 3294 (time 1779700394.91 total 50.09 last 57.85) - open/close 3752 (time 1779700404.94 total 60.12 last 45.66) - open/close 4215 (time 1779700414.94 total 70.12 last 46.28) total: 4500 open/close in 77.56 seconds: 58.02 ops/second - /unlink 1086 (time 1779700439.14 total 10.00 last 108.58) - /unlink 2130 (time 1779700449.14 total 20.00 last 104.37) - /unlink 3358 (time 1779700459.15 total 30.01 last 122.73) - /unlink 4342 (time 1779700469.18 total 40.04 last 98.08) total: 4500 /unlink in 41.77 seconds: 107.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 482 (time 1779700487.94 total 10.01 last 48.16) - open/close 992 (time 1779700497.94 total 20.01 last 50.99) - open/close 1618 (time 1779700507.95 total 30.01 last 62.57) - open/close 2206 (time 1779700517.96 total 40.03 last 58.74) - open/close 2996 (time 1779700527.96 total 50.03 last 78.99) - open/close 3535 (time 1779700537.98 total 60.04 last 53.81) - open/close 4172 (time 1779700547.99 total 70.06 last 63.61) total: 4500 open/close in 76.57 seconds: 58.77 ops/second - /unlink 1119 (time 1779700571.07 total 10.01 last 111.84) - /unlink 2365 (time 1779700581.09 total 20.02 last 124.42) - /unlink 3714 (time 1779700591.09 total 30.02 last 134.89) total: 4500 /unlink in 36.34 seconds: 123.85 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11196: 94217 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg132-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server unloading modules via unload_modules_local on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing unload_modules_local oleg132-server: modules unloaded.