== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:34:24 (1785090864) Stopping clients: oleg310-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg310-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg310-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg310-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg310-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg310-server oleg310-server: oleg310-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing load_modules_local oleg310-server: Loading modules from /home/green/git/lustre-release/lustre oleg310-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg310-server: detected 4 online CPUs by sysfs oleg310-server: MODOPTS_LIBCFS= oleg310-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg310-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg310-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 811 (time 1785091017.31 total 10.00 last 81.07) - open/close 1585 (time 1785091027.31 total 20.01 last 77.38) - open/close 2391 (time 1785091037.32 total 30.01 last 80.57) - open/close 3190 (time 1785091047.33 total 40.02 last 79.81) - open/close 3982 (time 1785091057.33 total 50.03 last 79.15) total: 4500 open/close in 56.49 seconds: 79.66 ops/second - /unlink 1581 (time 1785091077.83 total 10.00 last 158.08) - /unlink 2999 (time 1785091087.84 total 20.01 last 141.73) total: 4500 /unlink in 29.94 seconds: 150.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 804 (time 1785091111.27 total 10.02 last 80.24) - open/close 1623 (time 1785091121.27 total 20.02 last 81.88) - open/close 2419 (time 1785091131.28 total 30.02 last 79.59) - open/close 3229 (time 1785091141.28 total 40.03 last 80.92) - open/close 4044 (time 1785091151.29 total 50.04 last 81.47) total: 4500 open/close in 56.02 seconds: 80.33 ops/second - /unlink 1418 (time 1785091171.66 total 10.00 last 141.76) - /unlink 2702 (time 1785091181.67 total 20.01 last 128.35) - /unlink 4094 (time 1785091191.68 total 30.01 last 139.08) total: 4500 /unlink in 33.08 seconds: 136.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 770 (time 1785091209.65 total 10.00 last 76.96) - open/close 1549 (time 1785091219.66 total 20.02 last 77.81) - open/close 2352 (time 1785091229.67 total 30.02 last 80.23) - open/close 3189 (time 1785091239.69 total 40.05 last 83.49) - open/close 3980 (time 1785091249.69 total 50.05 last 79.06) total: 4500 open/close in 56.21 seconds: 80.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1410 (time 1785091270.19 total 10.01 last 140.91) - /unlink 2846 (time 1785091280.19 total 20.01 last 143.56) - /unlink 4231 (time 1785091290.20 total 30.01 last 138.44) total: 4500 /unlink in 32.36 seconds: 139.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 808 (time 1785091307.22 total 10.00 last 80.79) - open/close 1687 (time 1785091317.22 total 20.00 last 87.89) - open/close 2558 (time 1785091327.22 total 30.01 last 87.04) - open/close 3333 (time 1785091337.23 total 40.02 last 77.43) - open/close 4151 (time 1785091347.25 total 50.03 last 81.66) total: 4500 open/close in 54.54 seconds: 82.51 ops/second - /unlink 1334 (time 1785091366.35 total 10.01 last 133.32) - /unlink 2647 (time 1785091376.36 total 20.01 last 131.22) - /unlink 4098 (time 1785091386.36 total 30.01 last 145.06) total: 4500 /unlink in 32.94 seconds: 136.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 815 (time 1785091403.90 total 10.01 last 81.44) - open/close 1580 (time 1785091413.91 total 20.01 last 76.44) - open/close 2491 (time 1785091423.91 total 30.02 last 91.05) - open/close 3343 (time 1785091433.91 total 40.02 last 85.18) - open/close 4218 (time 1785091443.92 total 50.03 last 87.47) total: 4500 open/close in 53.61 seconds: 83.94 ops/second - /unlink 1466 (time 1785091461.21 total 10.01 last 146.45) - /unlink 2855 (time 1785091471.21 total 20.01 last 138.85) - /unlink 4390 (time 1785091481.22 total 30.02 last 153.45) total: 4500 /unlink in 30.94 seconds: 145.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 800 (time 1785091496.77 total 10.00 last 80.00) - open/close 1678 (time 1785091506.77 total 20.01 last 87.74) - open/close 2503 (time 1785091516.78 total 30.01 last 82.44) - open/close 3305 (time 1785091526.79 total 40.02 last 80.13) - open/close 4094 (time 1785091536.80 total 50.03 last 78.81) total: 4500 open/close in 55.48 seconds: 81.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1494 (time 1785091556.29 total 10.00 last 149.36) - /unlink 2964 (time 1785091566.29 total 20.01 last 146.93) - /unlink 4341 (time 1785091576.30 total 30.01 last 137.65) total: 4500 /unlink in 31.12 seconds: 144.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 788 (time 1785091591.17 total 10.01 last 78.71) - open/close 1515 (time 1785091601.17 total 20.02 last 72.66) - open/close 2339 (time 1785091611.19 total 30.03 last 82.27) - open/close 3146 (time 1785091621.20 total 40.04 last 80.64) - open/close 3952 (time 1785091631.20 total 50.04 last 80.56) total: 4500 open/close in 55.98 seconds: 80.38 ops/second - /unlink 1347 (time 1785091650.12 total 10.00 last 134.70) - /unlink 2703 (time 1785091660.12 total 20.00 last 135.57) - /unlink 4381 (time 1785091670.13 total 30.01 last 167.70) total: 4500 /unlink in 30.68 seconds: 146.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 865 (time 1785091685.26 total 10.00 last 86.47) - open/close 1680 (time 1785091695.27 total 20.01 last 81.44) - open/close 2546 (time 1785091705.28 total 30.02 last 86.48) - open/close 3339 (time 1785091715.28 total 40.03 last 79.29) - open/close 4125 (time 1785091725.29 total 50.04 last 78.51) total: 4500 open/close in 54.62 seconds: 82.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1480 (time 1785091743.55 total 10.00 last 147.99) - /unlink 2958 (time 1785091753.55 total 20.00 last 147.74) - /unlink 4360 (time 1785091763.55 total 30.01 last 140.14) total: 4500 /unlink in 31.31 seconds: 143.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 823 (time 1785091778.70 total 10.00 last 82.27) - open/close 1626 (time 1785091788.71 total 20.01 last 80.26) - open/close 2456 (time 1785091798.71 total 30.01 last 82.97) - open/close 3253 (time 1785091808.72 total 40.02 last 79.67) - open/close 4040 (time 1785091818.74 total 50.04 last 78.54) total: 4500 open/close in 56.31 seconds: 79.91 ops/second - /unlink 1395 (time 1785091838.88 total 10.00 last 139.47) - /unlink 2797 (time 1785091848.88 total 20.01 last 140.11) - /unlink 4205 (time 1785091858.89 total 30.01 last 140.75) total: 4500 /unlink in 32.21 seconds: 139.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 823 (time 1785091875.13 total 10.01 last 82.21) - open/close 1649 (time 1785091885.14 total 20.02 last 82.54) - open/close 2412 (time 1785091895.14 total 30.02 last 76.27) - open/close 3269 (time 1785091905.15 total 40.03 last 85.63) - open/close 4068 (time 1785091915.15 total 50.03 last 79.87) total: 4500 open/close in 55.15 seconds: 81.59 ops/second - /unlink 1607 (time 1785091933.55 total 10.01 last 160.50) - /unlink 3090 (time 1785091943.55 total 20.02 last 148.26) - /unlink 4496 (time 1785091953.57 total 30.03 last 140.40) total: 4500 /unlink in 30.08 seconds: 149.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 654 (time 1785091967.81 total 10.02 last 65.28) - open/close 1365 (time 1785091977.83 total 20.04 last 70.97) - open/close 2216 (time 1785091987.84 total 30.04 last 85.06) - open/close 2972 (time 1785091997.84 total 40.05 last 75.58) - open/close 3723 (time 1785092007.85 total 50.05 last 75.03) - open/close 4479 (time 1785092017.85 total 60.06 last 75.59) total: 4500 open/close in 60.28 seconds: 74.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1384 (time 1785092032.46 total 10.00 last 138.34) - /unlink 2799 (time 1785092042.46 total 20.01 last 141.48) - /unlink 4331 (time 1785092052.46 total 30.01 last 153.17) total: 4500 /unlink in 31.34 seconds: 143.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 777 (time 1785092067.84 total 10.00 last 77.69) - open/close 1554 (time 1785092077.86 total 20.02 last 77.52) - open/close 2353 (time 1785092087.87 total 30.04 last 79.79) - open/close 3154 (time 1785092097.91 total 40.07 last 79.80) - open/close 3956 (time 1785092107.93 total 50.09 last 80.07) total: 4500 open/close in 56.79 seconds: 79.24 ops/second - /unlink 1446 (time 1785092129.22 total 10.01 last 144.45) - /unlink 2648 (time 1785092139.23 total 20.01 last 120.19) - /unlink 4071 (time 1785092149.23 total 30.01 last 142.27) total: 4500 /unlink in 33.52 seconds: 134.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 725 (time 1785092166.58 total 10.00 last 72.48) - open/close 1442 (time 1785092176.62 total 20.04 last 71.46) - open/close 2184 (time 1785092186.65 total 30.06 last 74.00) - open/close 3001 (time 1785092196.65 total 40.06 last 81.68) - open/close 3832 (time 1785092206.66 total 50.07 last 83.04) total: 4500 open/close in 58.54 seconds: 76.87 ops/second - /unlink 1363 (time 1785092229.00 total 10.00 last 136.25) - /unlink 2815 (time 1785092239.00 total 20.01 last 145.17) - /unlink 4198 (time 1785092249.02 total 30.02 last 138.05) total: 4500 /unlink in 32.15 seconds: 139.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 826 (time 1785092265.07 total 10.03 last 82.35) - open/close 1585 (time 1785092275.07 total 20.03 last 75.89) - open/close 2418 (time 1785092285.07 total 30.03 last 83.30) - open/close 3226 (time 1785092295.07 total 40.04 last 80.75) - open/close 4011 (time 1785092305.08 total 50.04 last 78.46) total: 4500 open/close in 56.15 seconds: 80.14 ops/second - /unlink 1418 (time 1785092325.43 total 10.01 last 141.64) - /unlink 2748 (time 1785092335.43 total 20.01 last 132.98) - /unlink 4137 (time 1785092345.43 total 30.02 last 138.87) total: 4500 /unlink in 32.62 seconds: 137.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 810 (time 1785092362.64 total 10.01 last 80.95) - open/close 1569 (time 1785092372.66 total 20.03 last 75.69) - open/close 2372 (time 1785092382.67 total 30.04 last 80.25) - open/close 3185 (time 1785092392.68 total 40.05 last 81.23) - open/close 3966 (time 1785092402.69 total 50.06 last 78.03) total: 4500 open/close in 56.60 seconds: 79.50 ops/second - /unlink 1536 (time 1785092423.02 total 10.00 last 153.55) - /unlink 2996 (time 1785092433.02 total 20.01 last 145.97) - /unlink 4483 (time 1785092443.02 total 30.01 last 148.67) total: 4500 /unlink in 30.15 seconds: 149.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 828 (time 1785092456.97 total 10.00 last 82.76) - open/close 1577 (time 1785092466.98 total 20.01 last 74.84) - open/close 2383 (time 1785092476.99 total 30.02 last 80.52) - open/close 3179 (time 1785092487.01 total 40.04 last 79.45) - open/close 4048 (time 1785092497.01 total 50.04 last 86.87) total: 4500 open/close in 55.69 seconds: 80.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1528 (time 1785092516.25 total 10.00 last 152.79) - /unlink 2975 (time 1785092526.26 total 20.01 last 144.63) total: 4500 /unlink in 29.39 seconds: 153.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 805 (time 1785092549.50 total 10.02 last 80.34) - open/close 1633 (time 1785092559.51 total 20.02 last 82.78) - open/close 2460 (time 1785092569.51 total 30.03 last 82.65) - open/close 3265 (time 1785092579.51 total 40.03 last 80.49) - open/close 4120 (time 1785092589.52 total 50.04 last 85.40) total: 4500 open/close in 54.39 seconds: 82.73 ops/second - /unlink 1556 (time 1785092607.21 total 10.01 last 155.52) - /unlink 3193 (time 1785092617.21 total 20.01 last 163.69) total: 4500 /unlink in 28.82 seconds: 156.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 839 (time 1785092639.75 total 10.01 last 83.83) - open/close 1554 (time 1785092649.77 total 20.03 last 71.34) - open/close 2323 (time 1785092659.78 total 30.03 last 76.87) - open/close 3064 (time 1785092669.79 total 40.05 last 73.99) - open/close 3766 (time 1785092679.80 total 50.05 last 70.17) total: 4500 open/close in 59.78 seconds: 75.27 ops/second - /unlink 1500 (time 1785092703.00 total 10.00 last 149.95) - /unlink 3146 (time 1785092713.01 total 20.01 last 164.48) total: 4500 /unlink in 27.84 seconds: 161.63 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11243: 75387 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg310-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg310-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg310-server stop mds service on oleg310-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg310-server unloading modules via unload_modules_local on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing unload_modules_local oleg310-server: modules unloaded.