== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:02:59 (1787857379) Stopping clients: oleg448-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg448-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg448-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg448-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg448-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg448-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg448-server oleg448-server: oleg448-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg448-server' oleg448-server: oleg448-server.virtnet: executing load_modules_local oleg448-server: Loading modules from /home/green/git/lustre-release/lustre oleg448-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg448-server: detected 4 online CPUs by sysfs oleg448-server: MODOPTS_LIBCFS= oleg448-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg448-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg448-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg448-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 598 (time 1787857586.14 total 10.01 last 59.76) - open/close 1246 (time 1787857596.15 total 20.01 last 64.78) - open/close 1899 (time 1787857606.15 total 30.01 last 65.27) - open/close 2516 (time 1787857616.16 total 40.02 last 61.67) - open/close 3169 (time 1787857626.17 total 50.03 last 65.24) - open/close 3851 (time 1787857636.18 total 60.04 last 68.11) total: 4500 open/close in 69.58 seconds: 64.68 ops/second - /unlink 1317 (time 1787857660.62 total 10.00 last 131.69) - /unlink 2466 (time 1787857670.63 total 20.01 last 114.76) - /unlink 3627 (time 1787857680.63 total 30.01 last 116.08) total: 4500 /unlink in 37.82 seconds: 118.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 667 (time 1787857704.01 total 10.01 last 66.66) - open/close 1282 (time 1787857714.02 total 20.01 last 61.48) - open/close 1855 (time 1787857724.02 total 30.01 last 57.28) - open/close 2524 (time 1787857734.03 total 40.02 last 66.85) - open/close 3179 (time 1787857744.03 total 50.02 last 65.47) - open/close 3836 (time 1787857754.05 total 60.04 last 65.62) - open/close 4460 (time 1787857764.05 total 70.04 last 62.38) total: 4500 open/close in 70.55 seconds: 63.78 ops/second - /unlink 1309 (time 1787857779.60 total 10.01 last 130.83) - /unlink 2437 (time 1787857789.60 total 20.01 last 112.77) - /unlink 3628 (time 1787857799.60 total 30.01 last 119.05) total: 4500 /unlink in 36.97 seconds: 121.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 699 (time 1787857821.69 total 10.00 last 69.87) - open/close 1338 (time 1787857831.69 total 20.01 last 63.85) - open/close 1956 (time 1787857841.70 total 30.02 last 61.74) - open/close 2393 (time 1787857851.72 total 40.04 last 43.64) - open/close 2865 (time 1787857861.77 total 50.08 last 46.97) - open/close 3333 (time 1787857871.78 total 60.10 last 46.72) - open/close 3772 (time 1787857881.79 total 70.10 last 43.89) - open/close 4259 (time 1787857891.80 total 80.11 last 48.64) total: 4500 open/close in 84.03 seconds: 53.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1106 (time 1787857911.14 total 10.01 last 110.47) - /unlink 2283 (time 1787857921.14 total 20.02 last 117.66) - /unlink 3350 (time 1787857931.15 total 30.02 last 106.67) total: 4500 /unlink in 39.31 seconds: 114.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 598 (time 1787857955.52 total 10.00 last 59.79) - open/close 1182 (time 1787857965.55 total 20.03 last 58.24) - open/close 1825 (time 1787857975.57 total 30.05 last 64.15) - open/close 2432 (time 1787857985.59 total 40.07 last 60.61) - open/close 3079 (time 1787857995.60 total 50.08 last 64.64) - open/close 3732 (time 1787858005.63 total 60.10 last 65.13) - open/close 4356 (time 1787858015.63 total 70.11 last 62.37) total: 4500 open/close in 72.43 seconds: 62.13 ops/second - /unlink 1170 (time 1787858033.37 total 10.00 last 117.00) - /unlink 2326 (time 1787858043.37 total 20.00 last 115.60) - /unlink 3462 (time 1787858053.38 total 30.00 last 113.56) total: 4500 /unlink in 38.35 seconds: 117.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 603 (time 1787858077.16 total 10.01 last 60.23) - open/close 1126 (time 1787858087.16 total 20.01 last 52.30) - open/close 1631 (time 1787858097.16 total 30.01 last 50.49) - open/close 2116 (time 1787858107.18 total 40.02 last 48.44) - open/close 2671 (time 1787858117.18 total 50.03 last 55.46) - open/close 3289 (time 1787858127.21 total 60.06 last 61.63) - open/close 3971 (time 1787858137.21 total 70.06 last 68.20) total: 4500 open/close in 78.23 seconds: 57.52 ops/second - /unlink 1027 (time 1787858161.13 total 10.00 last 102.69) - /unlink 1990 (time 1787858171.14 total 20.01 last 96.24) - /unlink 2887 (time 1787858181.14 total 30.01 last 89.65) - /unlink 3929 (time 1787858191.15 total 40.01 last 104.18) total: 4500 /unlink in 45.20 seconds: 99.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 529 (time 1787858211.26 total 10.00 last 52.88) - open/close 1077 (time 1787858221.26 total 20.01 last 54.77) - open/close 1619 (time 1787858231.27 total 30.02 last 54.13) - open/close 2075 (time 1787858241.30 total 40.05 last 45.48) - open/close 2494 (time 1787858251.32 total 50.06 last 41.84) - open/close 3088 (time 1787858261.32 total 60.06 last 59.39) - open/close 3713 (time 1787858271.33 total 70.08 last 62.41) - open/close 4341 (time 1787858281.35 total 80.09 last 62.72) total: 4500 open/close in 82.56 seconds: 54.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1263 (time 1787858299.02 total 10.01 last 126.22) - /unlink 2332 (time 1787858309.03 total 20.01 last 106.85) - /unlink 3349 (time 1787858319.03 total 30.01 last 101.69) - /unlink 4431 (time 1787858329.04 total 40.02 last 108.12) total: 4500 /unlink in 40.97 seconds: 109.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 611 (time 1787858345.87 total 10.00 last 61.08) - open/close 1223 (time 1787858355.87 total 20.01 last 61.16) - open/close 1799 (time 1787858365.88 total 30.02 last 57.56) - open/close 2419 (time 1787858375.89 total 40.02 last 61.95) - open/close 3006 (time 1787858385.90 total 50.04 last 58.63) - open/close 3510 (time 1787858395.92 total 60.05 last 50.32) - open/close 4030 (time 1787858405.92 total 70.05 last 51.99) - open/close 4479 (time 1787858415.93 total 80.06 last 44.85) total: 4500 open/close in 80.56 seconds: 55.86 ops/second - /unlink 965 (time 1787858432.48 total 10.00 last 96.46) - /unlink 2212 (time 1787858442.48 total 20.01 last 124.67) - /unlink 3390 (time 1787858452.49 total 30.01 last 117.76) - /unlink 4441 (time 1787858462.50 total 40.03 last 104.93) total: 4500 /unlink in 40.69 seconds: 110.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 603 (time 1787858478.37 total 10.03 last 60.14) - open/close 1102 (time 1787858488.40 total 20.05 last 49.78) - open/close 1781 (time 1787858498.41 total 30.06 last 67.83) - open/close 2408 (time 1787858508.42 total 40.07 last 62.63) - open/close 2990 (time 1787858518.42 total 50.07 last 58.19) - open/close 3627 (time 1787858528.42 total 60.07 last 63.69) - open/close 4212 (time 1787858538.43 total 70.08 last 58.48) total: 4500 open/close in 74.45 seconds: 60.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1113 (time 1787858558.14 total 10.02 last 111.07) - /unlink 2237 (time 1787858568.14 total 20.02 last 112.40) - /unlink 3439 (time 1787858578.15 total 30.03 last 120.13) total: 4500 /unlink in 39.70 seconds: 113.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 641 (time 1787858602.92 total 10.00 last 64.08) - open/close 1224 (time 1787858612.92 total 20.01 last 58.26) - open/close 1874 (time 1787858622.92 total 30.01 last 65.00) - open/close 2532 (time 1787858632.94 total 40.03 last 65.68) - open/close 3176 (time 1787858642.94 total 50.03 last 64.38) - open/close 3815 (time 1787858652.96 total 60.04 last 63.82) - open/close 4417 (time 1787858662.97 total 70.06 last 60.10) total: 4500 open/close in 71.14 seconds: 63.25 ops/second - /unlink 1175 (time 1787858678.23 total 10.00 last 117.50) - /unlink 2348 (time 1787858688.23 total 20.01 last 117.22) - /unlink 3549 (time 1787858698.24 total 30.01 last 120.02) total: 4500 /unlink in 38.49 seconds: 116.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 493 (time 1787858722.14 total 10.01 last 49.27) - open/close 1047 (time 1787858732.14 total 20.01 last 55.37) - open/close 1612 (time 1787858742.15 total 30.02 last 56.45) - open/close 2168 (time 1787858752.16 total 40.03 last 55.54) - open/close 2714 (time 1787858762.17 total 50.04 last 54.54) - open/close 3338 (time 1787858772.18 total 60.05 last 62.36) - open/close 3991 (time 1787858782.18 total 70.05 last 65.26) total: 4500 open/close in 78.73 seconds: 57.16 ops/second - /unlink 1210 (time 1787858806.01 total 10.01 last 120.89) - /unlink 2271 (time 1787858816.02 total 20.03 last 105.92) - /unlink 3502 (time 1787858826.02 total 30.03 last 123.09) total: 4500 /unlink in 38.30 seconds: 117.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 721 (time 1787858849.20 total 10.00 last 72.08) - open/close 1387 (time 1787858859.21 total 20.01 last 66.55) - open/close 2072 (time 1787858869.22 total 30.02 last 68.41) - open/close 2722 (time 1787858879.23 total 40.03 last 64.93) - open/close 3343 (time 1787858889.24 total 50.04 last 62.04) - open/close 4017 (time 1787858899.25 total 60.05 last 67.35) total: 4500 open/close in 67.18 seconds: 66.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1343 (time 1787858920.81 total 10.00 last 134.24) - /unlink 2659 (time 1787858930.82 total 20.01 last 131.50) - /unlink 4020 (time 1787858940.82 total 30.01 last 136.10) total: 4500 /unlink in 33.67 seconds: 133.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 664 (time 1787858959.51 total 10.00 last 66.39) - open/close 1188 (time 1787858969.53 total 20.02 last 52.28) - open/close 1691 (time 1787858979.54 total 30.03 last 50.26) - open/close 2231 (time 1787858989.54 total 40.03 last 53.99) - open/close 2849 (time 1787858999.55 total 50.04 last 61.76) - open/close 3334 (time 1787859009.58 total 60.07 last 48.33) - open/close 3849 (time 1787859019.59 total 70.08 last 51.46) - open/close 4447 (time 1787859029.60 total 80.09 last 59.74) total: 4500 open/close in 81.04 seconds: 55.53 ops/second - /unlink 1250 (time 1787859045.85 total 10.00 last 125.00) - /unlink 2447 (time 1787859055.85 total 20.00 last 119.69) - /unlink 3646 (time 1787859065.85 total 30.00 last 119.90) total: 4500 /unlink in 37.74 seconds: 119.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 601 (time 1787859088.42 total 10.02 last 60.00) - open/close 1224 (time 1787859098.43 total 20.03 last 62.23) - open/close 1757 (time 1787859108.45 total 30.05 last 53.17) - open/close 2379 (time 1787859118.46 total 40.06 last 62.12) - open/close 2916 (time 1787859128.47 total 50.07 last 53.67) - open/close 3436 (time 1787859138.49 total 60.09 last 51.89) - open/close 4152 (time 1787859148.50 total 70.10 last 71.53) total: 4500 open/close in 75.39 seconds: 59.69 ops/second - /unlink 1295 (time 1787859169.11 total 10.01 last 129.43) - /unlink 2474 (time 1787859179.11 total 20.01 last 117.87) - /unlink 3703 (time 1787859189.11 total 30.01 last 122.89) total: 4500 /unlink in 37.04 seconds: 121.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 711 (time 1787859210.81 total 10.00 last 71.08) - open/close 1414 (time 1787859220.84 total 20.02 last 70.15) - open/close 2134 (time 1787859230.84 total 30.03 last 71.98) - open/close 2822 (time 1787859240.85 total 40.04 last 68.69) - open/close 3463 (time 1787859250.86 total 50.05 last 64.05) - open/close 4161 (time 1787859260.86 total 60.05 last 69.79) total: 4500 open/close in 64.87 seconds: 69.37 ops/second - /unlink 1222 (time 1787859280.12 total 10.02 last 122.01) - /unlink 2270 (time 1787859290.13 total 20.02 last 104.76) - /unlink 3561 (time 1787859300.13 total 30.02 last 129.04) total: 4500 /unlink in 37.53 seconds: 119.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 670 (time 1787859322.69 total 10.02 last 66.86) - open/close 1325 (time 1787859332.70 total 20.03 last 65.41) - open/close 2008 (time 1787859342.71 total 30.04 last 68.26) - open/close 2745 (time 1787859352.71 total 40.05 last 73.65) - open/close 3407 (time 1787859362.72 total 50.06 last 66.14) - open/close 4033 (time 1787859372.73 total 60.06 last 62.54) total: 4500 open/close in 66.80 seconds: 67.36 ops/second - /unlink 1377 (time 1787859393.79 total 10.00 last 137.69) - /unlink 2660 (time 1787859403.80 total 20.01 last 128.17) - /unlink 3937 (time 1787859413.81 total 30.02 last 127.60) total: 4500 /unlink in 34.64 seconds: 129.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 663 (time 1787859433.75 total 10.01 last 66.25) - open/close 1336 (time 1787859443.75 total 20.01 last 67.26) - open/close 2022 (time 1787859453.75 total 30.02 last 68.57) - open/close 2805 (time 1787859463.76 total 40.02 last 78.25) - open/close 3541 (time 1787859473.77 total 50.03 last 73.54) - open/close 4194 (time 1787859483.78 total 60.04 last 65.26) total: 4500 open/close in 64.80 seconds: 69.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 987 (time 1787859504.70 total 10.00 last 98.67) - /unlink 2228 (time 1787859514.70 total 20.01 last 124.07) - /unlink 3458 (time 1787859524.70 total 30.01 last 122.99) total: 4500 /unlink in 38.12 seconds: 118.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 743 (time 1787859547.04 total 10.01 last 74.24) - open/close 1378 (time 1787859557.05 total 20.02 last 63.41) - open/close 2036 (time 1787859567.05 total 30.03 last 65.77) - open/close 2698 (time 1787859577.06 total 40.04 last 66.14) - open/close 3389 (time 1787859587.09 total 50.06 last 68.94) - open/close 4033 (time 1787859597.12 total 60.09 last 64.21) total: 4500 open/close in 68.20 seconds: 65.98 ops/second - /unlink 1135 (time 1787859620.42 total 10.00 last 113.50) - /unlink 2258 (time 1787859630.42 total 20.00 last 112.29) - /unlink 3485 (time 1787859640.42 total 30.00 last 122.66) total: 4500 /unlink in 37.64 seconds: 119.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 541 (time 1787859663.12 total 10.01 last 54.05) - open/close 1166 (time 1787859673.14 total 20.03 last 62.36) - open/close 1725 (time 1787859683.14 total 30.04 last 55.88) - open/close 2338 (time 1787859693.17 total 40.07 last 61.12) - open/close 2891 (time 1787859703.18 total 50.08 last 55.25) - open/close 3530 (time 1787859713.21 total 60.10 last 63.73) - open/close 4146 (time 1787859723.22 total 70.11 last 61.55) total: 4500 open/close in 75.99 seconds: 59.22 ops/second - /unlink 1241 (time 1787859743.95 total 10.00 last 124.05) - /unlink 2490 (time 1787859753.95 total 20.01 last 124.85) - /unlink 3767 (time 1787859763.98 total 30.03 last 127.40) total: 4500 /unlink in 36.32 seconds: 123.91 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11369: 95375 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg448-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg448-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg448-server stop mds service on oleg448-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg448-server stop mds service on oleg448-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg448-server unloading modules via unload_modules_local on: 'oleg448-server' oleg448-server: oleg448-server.virtnet: executing unload_modules_local oleg448-server: modules unloaded.