== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:44:32 (1769078672) Stopping clients: oleg154-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg154-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg154-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg154-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg154-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg154-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg154-server oleg154-server: oleg154-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg154-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg154-server: detected 4 online CPUs by sysfs oleg154-server: MODOPTS_LIBCFS= oleg154-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg154-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg154-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg154-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 400 (time 1769078876.16 total 10.01 last 39.96) - open/close 937 (time 1769078886.16 total 20.01 last 53.69) - open/close 1623 (time 1769078896.17 total 30.01 last 68.58) - open/close 2263 (time 1769078906.18 total 40.03 last 63.93) - open/close 2936 (time 1769078916.19 total 50.04 last 67.22) - open/close 3552 (time 1769078926.20 total 60.05 last 61.53) - open/close 4160 (time 1769078936.21 total 70.05 last 60.77) total: 4500 open/close in 76.14 seconds: 59.10 ops/second - /unlink 1045 (time 1769078958.88 total 10.00 last 104.46) - /unlink 2003 (time 1769078968.88 total 20.01 last 95.77) - /unlink 3073 (time 1769078978.89 total 30.01 last 106.92) - /unlink 4165 (time 1769078988.90 total 40.02 last 109.12) total: 4500 /unlink in 44.08 seconds: 102.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 610 (time 1769079010.32 total 10.02 last 60.91) - open/close 1164 (time 1769079020.34 total 20.04 last 55.28) - open/close 1810 (time 1769079030.35 total 30.04 last 64.56) - open/close 2322 (time 1769079040.35 total 40.05 last 51.17) - open/close 2769 (time 1769079050.36 total 50.05 last 44.68) - open/close 3496 (time 1769079060.37 total 60.07 last 72.61) - open/close 4061 (time 1769079070.37 total 70.07 last 56.49) total: 4500 open/close in 77.32 seconds: 58.20 ops/second - /unlink 1164 (time 1769079093.24 total 10.00 last 116.37) - /unlink 2250 (time 1769079103.25 total 20.01 last 108.53) - /unlink 3013 (time 1769079113.27 total 30.03 last 76.12) - /unlink 3674 (time 1769079123.28 total 40.04 last 66.03) - /unlink 4438 (time 1769079133.30 total 50.06 last 76.23) total: 4500 /unlink in 51.35 seconds: 87.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 456 (time 1769079151.85 total 10.01 last 45.55) - open/close 1029 (time 1769079161.86 total 20.02 last 57.26) - open/close 1658 (time 1769079171.86 total 30.02 last 62.89) - open/close 2273 (time 1769079181.86 total 40.02 last 61.49) - open/close 2987 (time 1769079191.88 total 50.04 last 71.28) - open/close 3713 (time 1769079201.89 total 60.05 last 72.53) - open/close 4379 (time 1769079211.89 total 70.05 last 66.58) total: 4500 open/close in 71.63 seconds: 62.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1134 (time 1769079229.97 total 10.00 last 113.35) - /unlink 2275 (time 1769079239.97 total 20.01 last 114.05) - /unlink 3524 (time 1769079249.97 total 30.01 last 124.88) total: 4500 /unlink in 39.27 seconds: 114.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 468 (time 1769079274.84 total 10.02 last 46.72) - open/close 1095 (time 1769079284.85 total 20.02 last 62.65) - open/close 1677 (time 1769079294.86 total 30.04 last 58.13) - open/close 2180 (time 1769079304.87 total 40.04 last 50.27) - open/close 2905 (time 1769079314.87 total 50.05 last 72.45) - open/close 3559 (time 1769079324.88 total 60.05 last 65.39) - open/close 4314 (time 1769079334.88 total 70.05 last 75.49) total: 4500 open/close in 73.46 seconds: 61.25 ops/second - /unlink 1067 (time 1769079354.13 total 10.01 last 106.64) - /unlink 2322 (time 1769079364.13 total 20.01 last 125.42) - /unlink 3530 (time 1769079374.13 total 30.01 last 120.79) total: 4500 /unlink in 37.82 seconds: 118.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 776 (time 1769079397.68 total 10.01 last 77.53) - open/close 1536 (time 1769079407.68 total 20.01 last 75.98) - open/close 2282 (time 1769079417.69 total 30.01 last 74.59) - open/close 3001 (time 1769079427.70 total 40.03 last 71.80) - open/close 3748 (time 1769079437.70 total 50.03 last 74.67) total: 4500 open/close in 59.78 seconds: 75.27 ops/second - /unlink 1416 (time 1769079461.67 total 10.00 last 141.55) - /unlink 2530 (time 1769079471.68 total 20.01 last 111.37) - /unlink 3904 (time 1769079481.69 total 30.02 last 137.28) total: 4500 /unlink in 34.15 seconds: 131.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 722 (time 1769079500.79 total 10.00 last 72.19) - open/close 1454 (time 1769079510.80 total 20.01 last 73.14) - open/close 2197 (time 1769079520.80 total 30.02 last 74.26) - open/close 2970 (time 1769079530.81 total 40.02 last 77.27) - open/close 3723 (time 1769079540.81 total 50.02 last 75.27) - open/close 4429 (time 1769079550.82 total 60.03 last 70.58) total: 4500 open/close in 60.90 seconds: 73.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1233 (time 1769079566.10 total 10.01 last 123.21) - /unlink 2845 (time 1769079576.10 total 20.01 last 161.20) - /unlink 4306 (time 1769079586.10 total 30.01 last 146.06) total: 4500 /unlink in 31.59 seconds: 142.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 784 (time 1769079602.47 total 10.00 last 78.37) - open/close 1483 (time 1769079612.47 total 20.01 last 69.84) - open/close 2240 (time 1769079622.48 total 30.02 last 75.65) - open/close 3051 (time 1769079632.49 total 40.03 last 81.00) - open/close 3769 (time 1769079642.50 total 50.04 last 71.72) - open/close 4486 (time 1769079652.52 total 60.06 last 71.61) total: 4500 open/close in 60.31 seconds: 74.62 ops/second - /unlink 1102 (time 1769079667.32 total 10.01 last 110.10) - /unlink 2310 (time 1769079677.33 total 20.01 last 120.74) - /unlink 3529 (time 1769079687.33 total 30.02 last 121.87) total: 4500 /unlink in 36.82 seconds: 122.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 668 (time 1769079709.56 total 10.02 last 66.69) - open/close 1405 (time 1769079719.56 total 20.02 last 73.69) - open/close 2145 (time 1769079729.57 total 30.03 last 73.90) - open/close 2874 (time 1769079739.59 total 40.05 last 72.78) - open/close 3573 (time 1769079749.60 total 50.06 last 69.84) - open/close 4431 (time 1769079759.60 total 60.06 last 85.76) total: 4500 open/close in 60.94 seconds: 73.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1401 (time 1769079774.62 total 10.00 last 140.08) - /unlink 2713 (time 1769079784.63 total 20.00 last 131.17) - /unlink 3983 (time 1769079794.63 total 30.01 last 126.88) total: 4500 /unlink in 34.06 seconds: 132.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 792 (time 1769079814.33 total 10.01 last 79.15) - open/close 1535 (time 1769079824.33 total 20.01 last 74.28) - open/close 2387 (time 1769079834.34 total 30.02 last 85.12) - open/close 3145 (time 1769079844.34 total 40.02 last 75.78) - open/close 3923 (time 1769079854.35 total 50.03 last 77.78) total: 4500 open/close in 57.16 seconds: 78.73 ops/second - /unlink 1494 (time 1769079875.31 total 10.01 last 149.30) - /unlink 3003 (time 1769079885.32 total 20.01 last 150.79) - /unlink 4376 (time 1769079895.32 total 30.01 last 137.30) total: 4500 /unlink in 30.71 seconds: 146.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 801 (time 1769079910.54 total 10.01 last 80.06) - open/close 1597 (time 1769079920.54 total 20.01 last 79.56) - open/close 2327 (time 1769079930.56 total 30.03 last 72.89) - open/close 3195 (time 1769079940.57 total 40.04 last 86.69) - open/close 4065 (time 1769079950.58 total 50.05 last 86.93) total: 4500 open/close in 55.94 seconds: 80.44 ops/second - /unlink 1364 (time 1769079971.03 total 10.00 last 136.37) - /unlink 2888 (time 1769079981.04 total 20.01 last 152.23) - /unlink 4339 (time 1769079991.04 total 30.02 last 145.01) total: 4500 /unlink in 31.48 seconds: 142.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 903 (time 1769080006.73 total 10.00 last 90.28) - open/close 1755 (time 1769080016.74 total 20.01 last 85.11) - open/close 2544 (time 1769080026.75 total 30.02 last 78.86) - open/close 3449 (time 1769080036.75 total 40.03 last 90.42) - open/close 4329 (time 1769080046.76 total 50.03 last 87.97) total: 4500 open/close in 51.73 seconds: 86.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1591 (time 1769080062.12 total 10.01 last 158.98) - /unlink 3313 (time 1769080072.12 total 20.01 last 172.11) total: 4500 /unlink in 27.93 seconds: 161.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 886 (time 1769080093.77 total 10.00 last 88.56) - open/close 1783 (time 1769080103.77 total 20.01 last 89.64) - open/close 2660 (time 1769080113.77 total 30.01 last 87.69) - open/close 3473 (time 1769080123.78 total 40.01 last 81.28) - open/close 4427 (time 1769080133.78 total 50.01 last 95.40) total: 4500 open/close in 50.69 seconds: 88.77 ops/second - /unlink 1621 (time 1769080148.67 total 10.00 last 162.07) - /unlink 3116 (time 1769080158.67 total 20.01 last 149.43) total: 4500 /unlink in 29.54 seconds: 152.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 880 (time 1769080182.60 total 10.00 last 87.97) - open/close 1851 (time 1769080192.61 total 20.01 last 97.07) - open/close 2815 (time 1769080202.62 total 30.02 last 96.28) - open/close 3696 (time 1769080212.62 total 40.02 last 88.08) total: 4500 open/close in 48.95 seconds: 91.93 ops/second - /unlink 1651 (time 1769080235.39 total 10.01 last 165.00) - /unlink 3321 (time 1769080245.39 total 20.01 last 166.98) total: 4500 /unlink in 27.00 seconds: 166.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 853 (time 1769080265.62 total 10.00 last 85.26) - open/close 1703 (time 1769080275.62 total 20.00 last 85.00) - open/close 2562 (time 1769080285.62 total 30.01 last 85.88) - open/close 3352 (time 1769080295.63 total 40.01 last 78.94) - open/close 4157 (time 1769080305.63 total 50.01 last 80.50) total: 4500 open/close in 53.81 seconds: 83.63 ops/second - /unlink 1767 (time 1769080322.72 total 10.00 last 176.66) - /unlink 3266 (time 1769080332.72 total 20.01 last 149.85) total: 4500 /unlink in 27.76 seconds: 162.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 926 (time 1769080353.84 total 10.00 last 92.58) - open/close 1815 (time 1769080363.85 total 20.01 last 88.87) - open/close 2737 (time 1769080373.85 total 30.01 last 92.14) - open/close 3569 (time 1769080383.86 total 40.01 last 83.19) - open/close 4395 (time 1769080393.87 total 50.03 last 82.48) total: 4500 open/close in 51.48 seconds: 87.41 ops/second - /unlink 1521 (time 1769080409.69 total 10.00 last 152.07) - /unlink 3155 (time 1769080419.70 total 20.01 last 163.29) total: 4500 /unlink in 28.85 seconds: 155.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 1052 (time 1769080442.67 total 10.01 last 105.08) - open/close 2114 (time 1769080452.67 total 20.02 last 106.16) - open/close 3072 (time 1769080462.67 total 30.02 last 95.77) - open/close 4069 (time 1769080472.69 total 40.03 last 99.56) total: 4500 open/close in 45.23 seconds: 99.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1869 (time 1769080491.47 total 10.01 last 186.79) - /unlink 3726 (time 1769080501.47 total 20.01 last 185.69) total: 4500 /unlink in 23.92 seconds: 188.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 1023 (time 1769080518.61 total 10.00 last 102.30) - open/close 2046 (time 1769080528.61 total 20.00 last 102.29) - open/close 3073 (time 1769080538.63 total 30.02 last 102.52) - open/close 4196 (time 1769080548.63 total 40.02 last 112.27) total: 4500 open/close in 42.96 seconds: 104.74 ops/second - /unlink 2009 (time 1769080564.70 total 10.00 last 200.84) - /unlink 4194 (time 1769080574.70 total 20.00 last 218.50) total: 4500 /unlink in 21.62 seconds: 208.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 1115 (time 1769080589.27 total 10.00 last 111.50) - open/close 2100 (time 1769080599.27 total 20.00 last 98.47) - open/close 3249 (time 1769080609.28 total 30.01 last 114.85) - open/close 4369 (time 1769080619.28 total 40.01 last 111.97) total: 4500 open/close in 41.13 seconds: 109.42 ops/second - /unlink 2275 (time 1769080633.03 total 10.00 last 227.42) - /unlink 4363 (time 1769080643.03 total 20.01 last 208.69) total: 4500 /unlink in 20.62 seconds: 218.27 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11101: 94317 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg154-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg154-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg154-server stop mds service on oleg154-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg154-server stop mds service on oleg154-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg154-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing unload_modules_local oleg154-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg154-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded.