Use lateral join with subquery on user_statistics table in account_view for network stats

[sureshanaparti]

Description

This PR uses lateral join (introduced in MySQL 8.0.14) with subquery on user_statistics table in account_view for network stats.

There was a performance issue while querying account_view, which performs full table scans on the user_statistics table, resulting in execution time.

The original account_view used account_netstats_view which aggregates network statistics from the user_statistics table. MySQL's query optimizer could not push WHERE predicates through the nested view with GROUP BY, causing it to materialize the entire derived table before filtering. LATERAL JOIN (supported from MySQL 8.0.14+) allows correlated subqueries where the predicate can be applied before aggregation, avoiding materialization.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

• account_view updated with lateral join.

SHOW CREATE VIEW account_view;

• account_netstats_view shouldn't exist, this should fail.

SHOW CREATE VIEW account_netstats_view;

• account_view queries execute in less time than before (query performance improved).

SELECT * FROM account_view WHERE id = 3;

• EXPLAIN shows ref access on user_statistics (not index scan)

EXPLAIN ANALYZE SELECT * FROM account_view WHERE id = 3 \G

How did you try to break this feature and the system with this change?

[sureshanaparti]

@blueorangutan package

[blueorangutan]

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 17.62%. Comparing base (7324ef4) to head (c960fbf).

Additional details and impacted files

@@             Coverage Diff              @@
##               4.22   #12631      +/-   ##
============================================
- Coverage     17.62%   17.62%   -0.01%     
+ Complexity    15668    15665       -3     
============================================
  Files          5917     5917              
  Lines        531255   531255              
  Branches      64951    64951              
============================================
- Hits          93639    93616      -23     
- Misses       427077   427100      +23     
  Partials      10539    10539              

Flag	Coverage Δ
uitests	3.70% <ø> (ø)
unittests	18.69% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sureshanaparti]

@DaanHoogland @nvazquez & Others - Should we update MySQL version to 8.0.14 in the compatibility matrix here: https://docs.cloudstack.apache.org/en/4.22.0.0/releasenotes/compat.html ?

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16799

[nvazquez]

Thanks @sureshanaparti - actually the support for Mysql 8.4 is planned for 4.22.1 (#12417) as 8.0 is reaching EOL in April this year, so I think we can hold this PR until the 8.4 support is verified (and a doc PR won't be needed for 8.0.14 as the compatibility matrix will be updated to 8.4)

[DaanHoogland]

Thanks @sureshanaparti - actually the support for Mysql 8.4 is planned for 4.22.1 (#12417) as 8.0 is reaching EOL in April this year, so I think we can hold this PR until the 8.4 support is verified (and a doc PR won't be needed for 8.0.14 as the compatibility matrix will be updated to 8.4)

in that case we need to update the 4.20 branch as well.