== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:04:02 (1787583842) Stopping clients: oleg304-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg304-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg304-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg304-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg304-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg304-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg304-server oleg304-server: oleg304-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg304-server' oleg304-server: oleg304-server.virtnet: executing load_modules_local oleg304-server: Loading modules from /home/green/git/lustre-release/lustre oleg304-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg304-server: detected 4 online CPUs by sysfs oleg304-server: MODOPTS_LIBCFS= oleg304-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg304-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg304-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg304-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 629 (time 1787584063.74 total 10.02 last 62.76) - open/close 1236 (time 1787584073.75 total 20.03 last 60.63) - open/close 1901 (time 1787584083.77 total 30.05 last 66.39) - open/close 2576 (time 1787584093.77 total 40.05 last 67.49) - open/close 3260 (time 1787584103.78 total 50.07 last 68.30) - open/close 3843 (time 1787584113.79 total 60.08 last 58.25) - open/close 4394 (time 1787584123.84 total 70.13 last 54.83) total: 4500 open/close in 72.46 seconds: 62.10 ops/second - /unlink 1150 (time 1787584141.23 total 10.01 last 114.92) - /unlink 2419 (time 1787584151.23 total 20.01 last 126.87) - /unlink 3733 (time 1787584161.24 total 30.01 last 131.32) total: 4500 /unlink in 36.95 seconds: 121.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 609 (time 1787584183.07 total 10.03 last 60.73) - open/close 1212 (time 1787584193.09 total 20.05 last 60.19) - open/close 1793 (time 1787584203.10 total 30.06 last 58.00) - open/close 2431 (time 1787584213.12 total 40.08 last 63.72) - open/close 3063 (time 1787584223.12 total 50.08 last 63.19) - open/close 3710 (time 1787584233.12 total 60.08 last 64.68) - open/close 4391 (time 1787584243.14 total 70.10 last 67.95) total: 4500 open/close in 71.88 seconds: 62.60 ops/second - /unlink 1176 (time 1787584259.76 total 10.01 last 117.50) - /unlink 2266 (time 1787584269.76 total 20.01 last 108.99) - /unlink 3524 (time 1787584279.77 total 30.01 last 125.78) total: 4500 /unlink in 37.20 seconds: 120.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 515 (time 1787584302.55 total 10.00 last 51.48) - open/close 922 (time 1787584312.57 total 20.02 last 40.61) - open/close 1328 (time 1787584322.59 total 30.04 last 40.52) - open/close 1916 (time 1787584332.59 total 40.05 last 58.78) - open/close 2594 (time 1787584342.61 total 50.06 last 67.71) - open/close 3225 (time 1787584352.61 total 60.06 last 63.09) - open/close 3905 (time 1787584362.62 total 70.07 last 67.94) total: 4500 open/close in 78.99 seconds: 56.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1164 (time 1787584386.44 total 10.00 last 116.35) - /unlink 2501 (time 1787584396.45 total 20.01 last 133.64) - /unlink 3712 (time 1787584406.45 total 30.01 last 121.10) total: 4500 /unlink in 37.03 seconds: 121.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 711 (time 1787584428.59 total 10.00 last 71.07) - open/close 1380 (time 1787584438.61 total 20.02 last 66.80) - open/close 2074 (time 1787584448.61 total 30.03 last 69.34) - open/close 2802 (time 1787584458.64 total 40.05 last 72.61) - open/close 3560 (time 1787584468.64 total 50.06 last 75.79) - open/close 4216 (time 1787584478.64 total 60.06 last 65.59) total: 4500 open/close in 64.20 seconds: 70.09 ops/second - /unlink 1284 (time 1787584496.95 total 10.00 last 128.37) - /unlink 2512 (time 1787584506.95 total 20.00 last 122.79) - /unlink 3733 (time 1787584516.96 total 30.01 last 121.97) total: 4500 /unlink in 36.21 seconds: 124.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 722 (time 1787584537.01 total 10.01 last 72.15) - open/close 1406 (time 1787584547.02 total 20.02 last 68.28) - open/close 2051 (time 1787584557.03 total 30.03 last 64.49) - open/close 2704 (time 1787584567.04 total 40.04 last 65.19) - open/close 3321 (time 1787584577.05 total 50.05 last 61.67) - open/close 3969 (time 1787584587.06 total 60.06 last 64.71) total: 4500 open/close in 68.08 seconds: 66.10 ops/second - /unlink 1225 (time 1787584610.50 total 10.01 last 122.38) - /unlink 2449 (time 1787584620.51 total 20.01 last 122.37) - /unlink 3777 (time 1787584630.51 total 30.01 last 132.77) total: 4500 /unlink in 35.63 seconds: 126.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 704 (time 1787584650.62 total 10.01 last 70.34) - open/close 1393 (time 1787584660.63 total 20.01 last 68.87) - open/close 2112 (time 1787584670.63 total 30.02 last 71.83) - open/close 2824 (time 1787584680.64 total 40.03 last 71.17) - open/close 3617 (time 1787584690.64 total 50.03 last 79.27) - open/close 4316 (time 1787584700.66 total 60.05 last 69.77) total: 4500 open/close in 62.34 seconds: 72.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1328 (time 1787584717.46 total 10.01 last 132.67) - /unlink 2621 (time 1787584727.46 total 20.01 last 129.30) - /unlink 3996 (time 1787584737.46 total 30.01 last 137.48) total: 4500 /unlink in 34.32 seconds: 131.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 673 (time 1787584756.86 total 10.01 last 67.23) - open/close 1410 (time 1787584766.86 total 20.01 last 73.68) - open/close 2131 (time 1787584776.88 total 30.03 last 71.99) - open/close 2820 (time 1787584786.88 total 40.03 last 68.88) - open/close 3578 (time 1787584796.89 total 50.04 last 75.72) - open/close 4344 (time 1787584806.89 total 60.04 last 76.58) total: 4500 open/close in 62.11 seconds: 72.45 ops/second - /unlink 1427 (time 1787584822.70 total 10.01 last 142.61) - /unlink 2781 (time 1787584832.71 total 20.01 last 135.34) - /unlink 4135 (time 1787584842.71 total 30.01 last 135.36) total: 4500 /unlink in 32.88 seconds: 136.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 770 (time 1787584860.34 total 10.01 last 76.95) - open/close 1495 (time 1787584870.36 total 20.02 last 72.38) - open/close 2155 (time 1787584880.38 total 30.05 last 65.83) - open/close 2790 (time 1787584890.39 total 40.06 last 63.43) - open/close 3461 (time 1787584900.41 total 50.08 last 66.97) - open/close 4112 (time 1787584910.43 total 60.10 last 64.97) total: 4500 open/close in 65.79 seconds: 68.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1362 (time 1787584930.12 total 10.02 last 135.96) - /unlink 2536 (time 1787584940.13 total 20.02 last 117.36) - /unlink 3874 (time 1787584950.13 total 30.03 last 133.72) total: 4500 /unlink in 34.90 seconds: 128.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 664 (time 1787584969.28 total 10.01 last 66.35) - open/close 1461 (time 1787584979.28 total 20.01 last 79.68) - open/close 2134 (time 1787584989.30 total 30.03 last 67.16) - open/close 2861 (time 1787584999.31 total 40.03 last 72.69) - open/close 3589 (time 1787585009.32 total 50.05 last 72.72) - open/close 4330 (time 1787585019.32 total 60.05 last 74.09) total: 4500 open/close in 62.13 seconds: 72.43 ops/second - /unlink 1380 (time 1787585036.06 total 10.00 last 137.97) - /unlink 2906 (time 1787585046.07 total 20.01 last 152.48) - /unlink 4397 (time 1787585056.07 total 30.01 last 149.02) total: 4500 /unlink in 30.83 seconds: 145.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 715 (time 1787585070.34 total 10.01 last 71.41) - open/close 1429 (time 1787585080.34 total 20.02 last 71.37) - open/close 2159 (time 1787585090.35 total 30.03 last 72.92) - open/close 2881 (time 1787585100.36 total 40.04 last 72.13) - open/close 3610 (time 1787585110.37 total 50.04 last 72.86) - open/close 4273 (time 1787585120.39 total 60.06 last 66.18) total: 4500 open/close in 63.63 seconds: 70.72 ops/second - /unlink 1323 (time 1787585138.39 total 10.00 last 132.30) - /unlink 2598 (time 1787585148.40 total 20.00 last 127.44) - /unlink 3909 (time 1787585158.40 total 30.01 last 131.09) total: 4500 /unlink in 34.46 seconds: 130.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 673 (time 1787585177.53 total 10.01 last 67.25) - open/close 1364 (time 1787585187.54 total 20.01 last 69.07) - open/close 2040 (time 1787585197.55 total 30.02 last 67.51) - open/close 2762 (time 1787585207.56 total 40.04 last 72.12) - open/close 3447 (time 1787585217.57 total 50.04 last 68.45) - open/close 4184 (time 1787585227.58 total 60.06 last 73.61) total: 4500 open/close in 64.84 seconds: 69.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1288 (time 1787585247.48 total 10.04 last 128.35) - /unlink 2321 (time 1787585257.48 total 20.04 last 103.25) - /unlink 3490 (time 1787585267.48 total 30.04 last 116.90) total: 4500 /unlink in 38.20 seconds: 117.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 607 (time 1787585291.35 total 10.00 last 60.68) - open/close 1213 (time 1787585301.37 total 20.02 last 60.48) - open/close 1888 (time 1787585311.38 total 30.03 last 67.42) - open/close 2555 (time 1787585321.39 total 40.04 last 66.63) - open/close 3268 (time 1787585331.39 total 50.05 last 71.28) - open/close 3916 (time 1787585341.40 total 60.05 last 64.79) - open/close 4474 (time 1787585351.41 total 70.06 last 55.73) total: 4500 open/close in 70.34 seconds: 63.98 ops/second - /unlink 1285 (time 1787585367.45 total 10.00 last 128.47) - /unlink 2666 (time 1787585377.46 total 20.01 last 138.06) - /unlink 3995 (time 1787585387.46 total 30.01 last 132.82) total: 4500 /unlink in 34.08 seconds: 132.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 713 (time 1787585406.21 total 10.01 last 71.25) - open/close 1425 (time 1787585416.22 total 20.02 last 71.09) - open/close 2183 (time 1787585426.22 total 30.03 last 75.77) - open/close 2904 (time 1787585436.23 total 40.04 last 72.03) - open/close 3649 (time 1787585446.24 total 50.04 last 74.44) - open/close 4330 (time 1787585456.24 total 60.04 last 68.09) total: 4500 open/close in 62.62 seconds: 71.87 ops/second - /unlink 1198 (time 1787585473.35 total 10.00 last 119.76) - /unlink 2498 (time 1787585483.35 total 20.01 last 129.97) - /unlink 3796 (time 1787585493.35 total 30.01 last 129.71) total: 4500 /unlink in 35.13 seconds: 128.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 681 (time 1787585513.34 total 10.01 last 68.04) - open/close 1395 (time 1787585523.34 total 20.01 last 71.39) - open/close 2048 (time 1787585533.34 total 30.01 last 65.29) - open/close 2701 (time 1787585543.35 total 40.01 last 65.29) - open/close 3414 (time 1787585553.36 total 50.03 last 71.21) - open/close 4064 (time 1787585563.38 total 60.04 last 64.88) total: 4500 open/close in 67.11 seconds: 67.05 ops/second - /unlink 1144 (time 1787585585.88 total 10.00 last 114.35) - /unlink 2408 (time 1787585595.88 total 20.01 last 126.35) - /unlink 3664 (time 1787585605.88 total 30.01 last 125.60) total: 4500 /unlink in 37.06 seconds: 121.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 564 (time 1787585628.01 total 10.01 last 56.37) - open/close 1111 (time 1787585638.02 total 20.01 last 54.70) - open/close 1684 (time 1787585648.02 total 30.01 last 57.27) - open/close 2230 (time 1787585658.05 total 40.04 last 54.44) - open/close 2881 (time 1787585668.06 total 50.05 last 65.03) - open/close 3574 (time 1787585678.07 total 60.06 last 69.24) - open/close 4181 (time 1787585688.09 total 70.08 last 60.58) total: 4500 open/close in 76.27 seconds: 59.00 ops/second - /unlink 1161 (time 1787585710.09 total 10.00 last 116.05) - /unlink 2347 (time 1787585720.09 total 20.01 last 118.54) - /unlink 3536 (time 1787585730.10 total 30.01 last 118.84) total: 4500 /unlink in 39.34 seconds: 114.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 549 (time 1787585755.53 total 10.00 last 54.88) - open/close 1137 (time 1787585765.54 total 20.01 last 58.75) - open/close 1720 (time 1787585775.56 total 30.03 last 58.20) - open/close 2291 (time 1787585785.56 total 40.03 last 57.09) - open/close 2891 (time 1787585795.58 total 50.05 last 59.85) - open/close 3373 (time 1787585805.60 total 60.07 last 48.13) - open/close 3929 (time 1787585815.61 total 70.08 last 55.51) total: 4500 open/close in 78.96 seconds: 56.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1154 (time 1787585838.42 total 10.01 last 115.34) - /unlink 2471 (time 1787585848.43 total 20.02 last 131.56) - /unlink 3546 (time 1787585858.43 total 30.02 last 107.47) total: 4500 /unlink in 39.39 seconds: 114.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 608 (time 1787585883.86 total 10.02 last 60.68) - open/close 1164 (time 1787585893.86 total 20.02 last 55.57) - open/close 1834 (time 1787585903.87 total 30.03 last 66.97) - open/close 2534 (time 1787585913.87 total 40.03 last 69.97) - open/close 3250 (time 1787585923.88 total 50.04 last 71.53) - open/close 3928 (time 1787585933.89 total 60.05 last 67.76) total: 4500 open/close in 68.45 seconds: 65.75 ops/second - /unlink 1229 (time 1787585956.78 total 10.00 last 122.88) - /unlink 2429 (time 1787585966.78 total 20.00 last 119.98) - /unlink 3733 (time 1787585976.78 total 30.01 last 130.34) total: 4500 /unlink in 35.45 seconds: 126.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 731 (time 1787585997.05 total 10.02 last 72.98) - open/close 1471 (time 1787586007.05 total 20.02 last 73.97) - open/close 2144 (time 1787586017.05 total 30.02 last 67.27) - open/close 2849 (time 1787586027.06 total 40.03 last 70.42) - open/close 3385 (time 1787586037.07 total 50.04 last 53.58) - open/close 3946 (time 1787586047.07 total 60.04 last 56.07) total: 4500 open/close in 69.51 seconds: 64.74 ops/second - /unlink 1308 (time 1787586072.18 total 10.00 last 130.77) - /unlink 2446 (time 1787586082.19 total 20.01 last 113.67) - /unlink 3692 (time 1787586092.19 total 30.02 last 124.57) total: 4500 /unlink in 36.14 seconds: 124.53 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95254 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg304-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg304-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg304-server stop mds service on oleg304-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg304-server stop mds service on oleg304-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg304-server unloading modules via unload_modules_local on: 'oleg304-server' oleg304-server: oleg304-server.virtnet: executing unload_modules_local oleg304-server: modules unloaded.