8000 Core improvements and new features by markoceri · Pull Request #19 · edge-mining/core · GitHub
[go: up one dir, main page]

Skip to content

Core improvements and new features - #19

Merged
GitGab19 merged 96 commits into
edge-mining:mainfrom
markoceri:basic-engine-rule
Aug 28, 2025
Merged

GitGab19 merged 96 commits into
edge-mining:mainfrom
markoceri:basic-engine-rule

Conversation

@markoceri
Copy link
Copy Markdown
Collaborator

The Edge Mining system now has his (basic) Rule Engine that allows users to define custom automation rules to control when mining should start or stop, based on the system's energy status. Additionally, support for storing optimization policies as YAML files instead of SQLite database records has been added. This provides a more readable and intuitive way to manage mining policies. Finally, the Edge Mining project is now fully configured with all the development tools needed to maintain high code quality.

Main changes include:

Edge Mining Rule Engine and YAML Policy System

Condition Types

The rule engine supports various condition types to create complex logical rules for mining control. Single Conditions can be used for simple checks, while Multiple Logical Conditions allow for more sophisticated rules. Multiple conditions support logical operators AND, OR, and NOT to combine conditions in flexible ways.

Single Condition

The simplest form of a condition checks a single field against a value. It must be true for the rule to apply.

conditions:
  field: "energy_state.battery.state_of_charge"
  operator: "gt"
  value: 80

Multiple Logical Conditions

A more complex rule can combine multiple conditions using logical operators. This allows for sophisticated decision-making based on the system's state.

AND: All the conditions must be true for the rule to apply.

conditions:
  all_of:
    - field: "energy_state.battery.state_of_charge"
      operator: "gt"
      value: 50
    - field: "forecast.next_hour_power"
      operator: "gt"
      value: 1500

OR: At least one condition must be true for the rule to apply.

conditions:
  any_of:
    - field: "energy_state.battery.state_of_charge"
      operator: "lt"
      value: 20
    - field: "energy_state.grid.importing_power"
      operator: "gt"
      value: 3000

NOT: This negates a condition, meaning the rule applies if the condition is false.

conditions:
  not_:
    field: "miner.status"
    operator: "eq"
    value: "STARTING"

We can also combine multiple conditions using logical operators:

Complex Combinations:

conditions:
  all_of:
    - field: "timestamp.weekday"
      operator: "in"
      value: [5, 6]  # Weekend
    - any_of:
        - field: "energy_state.battery.state_of_charge"
          operator: "gt"
          value: 90
        - field: "energy_state.grid.exporting_power"
          operator: "lt"
          value: 1000

Supported Operators

It's a basic rule engine, but it has some great superpowers. It supports all the basic logical operators and more.

Operator Description Example
eq Equal value: 50
ne Not equal value: "ERROR"
gt Greater than value: 80
gte Greater than or equal value: 90
lt Less than value: 20
lte Less than or equal value: 10
in Contained in list value: [5, 6, 0]
not_in Not contained in list value: [1, 2, 3]
contains String contains value: "solar"
starts_with String starts with value: "HIGH"
ends_with String ends with value: "_LEVEL"
regex Regular expression value: "^[0-9]+$"

Available Fields in DecisionalContext

All the entities in the DecisionalContext can be used as fields in conditions with the dot notation. This includes:

  • energy_state.battery.state_of_charge
  • energy_state.battery.current_power
  • forecast.next_hour_power
  • energy_state.grid.exporting_power
  • etc.

Directory Structure

Policies are now stored in the optimization_policies/ directory in the application root. In the rule_examples/ directory, you can find example rules that demonstrate the rule engine capabilities (this directory is not used by the system, but it is useful only as examples).

core/
├── optimization_policies/
│   ├── 550e8400-e29b-41d4-a716-446655440000.yaml
│   └── another-policy-id.yaml
├── rules_examples/
│   ├── start/
|   │   ├── advanced_rules.yaml
|   │   └── basic_rules.yaml
│   └── stop/
|       ├── advanced_rules.yaml
|       └── basic_rules.yaml
├── edgemining.db (still used for other data)
└── ...

Be curious! You can explore the rules_examples/ directory to see how the rule engine works and how to create your own rules 😊.

Policy File Format

Each policy is stored as a separate YAML file named with the policy ID. The format includes:

id: "550e8400-e29b-41d4-a716-446655440000"
name: "Advanced Solar Mining Policy"
description: "Comprehensive policy for solar-based mining optimization with intelligent start and stop conditions"

start_rules:
  - id: "550e8400-e29b-41d4-a716-446655440001"
    name: "Priority Solar Mining"
    description: "High priority: Start mining when solar production is very high"
    conditions:
      all_of:
        - field: "energy_state.production"
          operator: "gt"
          value: 3000  # Very high solar production
        - field: "energy_state.battery.state_of_charge"
          operator: "gt"
          value: 40   # Battery not too low
    priority: 20
    enabled: true

stop_rules:
  - id: "550e8400-e29b-41d4-a716-446655440004"
    name: "Critical Battery Protection"
    description: "CRITICAL: Stop mining immediately when battery is critically low"
    conditions:
      any_of:
        - field: "energy_state.battery.state_of_charge"
          operator: "lt"
          value: 15  # Critical battery level
        - field: "energy_state.battery.remaining_capacity"
          operator: "lt"
          value: 800  # Less than 800 WattHours remaining
        - all_of:  # Battery discharging fast + low SOC
            - field: "energy_state.battery.discharging_power"
              operator: "lt"
              value: 1000  # Discharging more than 1kW
            - field: "energy_state.battery.state_of_charge"
              operator: "lt"
              value: 25
    priority: 100  # Highest priority for safety
    enabled: true

metadata:
  version: 2
  author: "Edge Mining User"
  created: "2025-08-04"
  last_modified: "2025-08-04"

Best Practices

  1. Priority: Assign higher priorities to safety rules (stop for low battery).
  2. Descriptive Names: Use clear and descriptive rule and policy names.
  3. Descriptions: Always add descriptions to document behavior.
  4. Versioning: Use metadata to track configuration versions (it is automatically added by the system).

Hybrid Persistence

The YAML adapter uses a hybrid approach:

  • Policies: Stored as YAML files (human-readable, version-controllable).
  • Other data: Still stored in SQLite (miners, energy sources, etc.).

Development Tools

The Edge Mining project is now fully configured with all the necessary development tools to maintain high code quality. Most of them are included as pre-commit hooks, so they run automatically before each commit. (It is recommended to install pre-hooks before committing. 😉).

Configured Tools

Code Formatting

  • Ruff: Linter and formatter for Python.

Quality Control

  • bandit: Code security analysis.
  • mypy: Static type checking (configured but not mandatory).
  • yamllint: YAML file syntax checking.

Testing

  • pytest: Testing framework.
  • pytest-cov: Test coverage.

Pre-commit Hooks

  • Automatic execution of all checks before each commit.
  • Configuration in .pre-commit-config.yaml with all updated versions.

Configuration Files

Main

  • .pre-commit-config.yaml: Pre-commit hooks.
  • pyproject.toml: Ruff, mypy and Bandit configuration.
  • requirements-dev.txt: Development dependencies.
  • Makefile: Automation commands.

VS Code

  • .vscode/settings.json: Recommended settings.
  • .vscode/extensions.json: Recommended extensions.
  • .vscode/launch.json: Debug configurations.

Documentation

  • DEVELOPMENT_WORKFLOW.md: Development workflow guide.
  • DEV_TOOLS.md: Development tools documentation.

Useful Scripts

  • Makefile: Automation with make (recommended).

Quick Commands

Initial Setup

make setup

Daily Use

make format    # Format the code
make lint      # Quality check
make test      # Run tests
make pre-commit # Complete pre-commit check

Automation

Pre-commit hooks run automatically on every git commit.

Special Features

  • Permissive Configuration: ruff configured to be less strict during development.
  • Optional mypy: Type checking configured but not mandatory for commits.
  • VS Code Ready: Optimized settings for VS Code.
  • Complete Makefile: All necessary commands automated.
  • Complete Documentation: Detailed guides for developers.

Benefits Achieved

  1. Version Control: Default policies can be tracked in Git as YAML files.
  2. Human Readable: Policies and Rules are easy to review and edit.
  3. Backup: Simple file-based backup and restore.
  4. Sharing: Easy to share policies between users (What do you think about a cool policy repository to download the rules?).
  5. Code Quality: Automatic checks on formatting, style and security.
  6. Consistency: All developers follow the same standards.
  7. Automation: Many checks and corrections are automatic.
  8. Productivity: Less time spent on manual code reviews.
  9. Security: Automatic security checks with bandit.
  10. Testing: Ready testing framework with coverage.

Additional Features

  • All the entities can now be configured by CLI interactive menu using the command: python3 -m edge_mining cli interactive.
  • A Sun entity has been added to the Energy Forecast domain, which provides the data about the time and position of the sun in the sky that can be used in the rules (eg: sunrise, sunset, azimuth, elevation, etc.).

And that's it! The Edge Mining project is now ready for development with all quality tools configured and working, a tiny rule engine to control the miners and a lot of stranded energy to mine! 😄

If you're a renewable energy plant owner, I hope you mine your energy. If you're a developer, I hope you contribute to the project. But... if you're both, you hold the keys to the future.
Happy coding! 🚀

Marco

markoceri added 30 commits May 20, 2025 18:23
…Edge Mining application

- Implemented configurations for home load, miner, notification, and performance tracking adapters.
- Created mapping files for energy, external services, forecast, home load, miner, notification, and performance tracker adapters.
- Introduced common objects, entities, exceptions, and ports for external services.
- Refactored settings to streamline adapter configuration and added MQTT support (to be completed).
- Updated requirements to include loguru and paho-mqtt for enhanced logging and MQTT capabilities.
…nd update references with a ServiceContainer
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

API data models are not matching the domain data model

2 participants

0