Quarterly announce about new features

[evyncke]

I have seen email dated 6th of June 2022 about 'YANG Catalog 5.3.0 release', and this is really a good initiative. Same for the CHANGELOG.md (e.g., https://github.com/YangCatalog/backend/blob/master/CHANGELOG.md for release 5.4.0)

It needs to be continued at least on a quarterly basis (or when a major feature has been added). Then, the IETF tools team will forward this email to some IETF mailing lists that could be interested (e.g., opsawg@ietf.org and/or tools-discuss@ietf.org)

[SlavomirMazurPantheon]

Hello @evyncke,

current plan is to sent informative email after each Release with info about the latest improvements/changes/fixes.
Sending email after each quartal which will summarize changes in whole quartal is also something which I was thinking about.

I believe I can put it together after Q3.

[evyncke]

Superb, thank you

[SlavomirMazurPantheon]

2022 Q3 Summary

Matomo Analytics:

• Matomo Analytics tools in now implemented into YANG Catalog. It track YC page visits and also API usage.
• Every Monday morning, a report for the previous is sent to an e-mail

Problematic IETF drafts:

• We have identified IETF drafts where the xym tool reports an error
• List of these drafts is displayed here: https://yangcatalog.org/private-page?tab=IETF
• Initially, xym reported an error on some drafts, even though it was not a real problem
• This resulted in several PRs done by Richard and new release of the xym tool - 0.6.0
• We also implemented notifying the authors of these drafts (with the option to unsubscribe from these email), this will be working very soon

Github Actions/Testing:

• Github Actions pipeline was set in sdo-analysis - it runs tests everytime some push changes to the repo
• 9 Python scripts from sdo-analysis are now fully covered with unittests (more tests)
• Number of existing tests have been improved in the backend repository
• Tests have already helped us to find some bugs several times and are important part of our development
• ycclient compatibility re-tested

Code refactoring:

• huge amount of code has been refactored (mostly in backend and sdo-analysis repos)
  • e.g: using f-strings, removing bare 'except:' blocks, moving functionality into methods, creating classes
• the main reason is to improve readability and testability of these scripts
• this already allowed us to test some of the scripts in sdo-analysis

Yang Catalog Github:

• web_root, admin_ui and yangcatalog-ui repositories archived after merging all into single frontend repository
• Overview section added to README.md files in all repositories
• We are trying to link each PR to some Github issue with a detailed description of bug/feature
• since July we started using Conventional Commits for commit messages
• we started with "regular" releases where we summarize all the improvements on an approximately monthly basis

Other:

• we try to keep the versions of used tools (pyang, confd ...) and dependencies as up-to-date as possible
• Also if needed we are looking for new/better pypi packages or removing unnecessary ones
• Since the number of populated modules is still growing, we are trying to find all possible improvements to speed up the populating process and also improve cronjobs which are running on daily or weekly basis.
• You can display list of implementations (with affected modules) in Module details now

New developers:

• I created a setup.sh script which, when run on a clean Linux environment, automatically creates the folder structure and the files necessary to run the project
• In multiple scripts, we also covered a number of edge-cases that can occur when running fresh environment (e.g: missing files/folders)
• Steps required to run the YC locally have been written and updated several times
• This script was also used by two new developers who are currently helping us at YC

[SlavomirMazurPantheon]

2022 Q4 Summary

Tools Upgrades

• Python base image bumped to version 3.10 in multiple repos
• PROD server updated to Ubuntu 22.10
• ConfD updated to version 8.0
• xym tool updated to version 0.6.2
• yanglint updated to version v2.1.4

Code refactoring:

• STYLEGUIDE.md file created
• Python code reformatted according to the defined STYLEGUIDE.md in all repositories
• Functionality for fetching all modules unified across YC
• Multiple scripts refactored in modules-compilation repository

Github Actions/Testing:

• Code Coverage is now automatically generated with each commit
• Code Coverage badge is generated and added to the README.md file after merging Pull Request
• Unit tests covering all the module property resolvers added - backend repository
• check_archived_drafts.py script completelly covered by unit tests

YANG Validator

• Autocomplete functionality added for IETF draft names in UI form

YANG Search

• Possibility to enable/disable usage of synonyms added
• Text search (grep) functionality implementation (currently still in testing phase)
• Incorrect use of Elasticsearch scroll fixed
• API Endpoints moved to separate files to reduce yangSearch.py and ycSearch.py files (reducing similiar filenames)

Other improvements

• Module implementations added without need of re-parsing whole module
• Script for resetting a forgotten password added
• sdo_analysis repository renamed to module-compilation
• Send email notification if jobs (Github Actions) failed in YangModels/yang repository
• Multiple improvements to the logging added

Documentation

• Multiple README.md files added in directories to better explain meaning of individual files/scripts