Load config, harden processing and notifications

- Actually load configs/config.yaml at startup (init_config was commented out)
- Add timeouts and error handling to all HTTP calls; run blocking requests
  in threads; serialize processing with a lock; run initial check in background
- URL-encode Telegram messages and HTML-escape scraped data
- Fix GPV crash on empty schedule, 00:00 end time, date sorting and fallback format
- Parse PO table per row; fix daily reminder check and change detection
- Use per-instance timestamps in dataclasses; robust config value parsing
- Matrix: use v3 PUT send endpoint; strip secrets from logged errors
- Resolve templates/static/test paths relative to the project; use lifespan
- Move debug GPV data to test/data_gpv.json; add CLAUDE.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Anry Das
2026-10-13 14:04:18 +03:00
co-authored by Claude Opus 5.5
parent e3367c6a2e
commit 22fc8b38bf
8 changed files with 683 additions and 444 deletions
+3
View File
@@ -1 +1,4 @@
/configs/secrets.yaml /configs/secrets.yaml
__pycache__/
.idea/
.venv/
+49
View File
@@ -0,0 +1,49 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## What this is
FastAPI service that polls the Ivano-Frankivsk energy company (oe.if.ua) for power outages at a configured address/account and pushes notifications to Telegram and (optionally) Matrix. Two independent data sources:
- **PO** (planned outages) — HTML table scraped from `https://oe.if.ua/uk/shutdowns_table`, parsed with lxml XPath per row (`div[@class='table-row flex']`).
- **GPV** (hourly outage schedule) — JSON from `https://be-svitlo.oe.if.ua/schedule-by-search` by account number (requested with `verify=False`).
Messages and log text are in Ukrainian; keep that.
## Commands
```bash
.venv/bin/pip install -r requirements.txt
.venv/bin/python main.py # runs uvicorn with host/port from configs/config.yaml
```
There is no test suite or linter configured. `test/` holds fixtures only (`data_po_2days_*.html`, `data_gpv.json`). To exercise the app without contacting real sites or messengers, set `application.debug: "true"` in `configs/config.yaml`: fetches then read the fixtures, and senders `print()` instead of sending. Note fixtures are dated April 2026, so entries are pruned as expired unless dates are shifted.
## Configuration
- `configs/config.yaml` is loaded at import of `main.py` via `cfg.init_config()`; values use `!secret <key>` resolved from `configs/secrets.yaml` (gitignored, must exist for a real run). The `!secret` tag is registered on a private loader subclass in `config_file.read_yaml`.
- `app_config.py` exposes settings as **module-level globals** (`cfg.TG_TOKEN`, `cfg.IS_DEBUG`, …) populated by `parse_config()`. Always access them as `cfg.X` at call time — `from app_config import X` would capture the pre-config default.
- Booleans in YAML may be strings (`"true"`) — parse with `as_bool`.
- `configs/log.yaml` is the uvicorn log config. All modules log through the single `cfg.LOG` logger, which also feeds the in-memory `LOG_LIST` served by `/api/v1/logs` (unauthenticated — never log secrets; use `utils.safe_error(e)` for request exceptions, since TG URLs contain the token).
## Architecture
- `main.py` — FastAPI `app` with a sub-app `api` mounted at `/api/v1` (the root HTML page in `templates/info.html` lists `api` routes). `lifespan` creates an `AsyncIOScheduler` with one cron job per (hour in `[start_hour, stop_hour)`, minute in `PROCESSING_MINUTES`) — every 2 minutes in debug — and runs an initial check as a background task. `check_and_notify()` runs `process_gpv` then `process_po` under `PROCESS_LOCK` so scheduled runs and `/update` never overlap (prevents duplicate notifications).
- `utils.py` — fetching, parsing, state and notification. State is in-memory only (`PARSED_PO_DATAS`, `PARSED_GPV_DATAS`); lost on restart, so everything current is re-notified after a restart. Blocking `requests` calls are run via `asyncio.to_thread`; every request has `cfg.HTTP_TIMEOUT`; `process_*` catch network/parse errors and return status codes instead of raising.
- `model.py` — `POData` / `GPVData` dataclasses. Service fields (`notified_at`, `updated_at`, `need_to_send`, `start_ts`) are `compare=False`, so `==` means "same outage content"; this drives change detection.
### Notification lifecycle (per poll)
1. Parse fresh entries → `update_*_data` merges into the list by `key` (PO: start ts + address; GPV: event date + queue).
- not yet sent → replaced with fresh data;
- already sent and content changed (PO `!=`, GPV `is_same_schedule`) → replaced, so it is re-sent;
- PO only: also re-sent by `is_reminder_needed` (a day since last notification, or the first processing slot of the day passed).
2. `remove_old_*_data` prunes expired entries (PO: 1h after start; GPV: 30 min after the last power-on / end of day). GPV `start_ts` holds power-**on** timestamps despite the name; `00:00` end rolls to the next day.
3. `check_and_send_*` sends entries with `need_to_send` via `notify()`, which sends to TG and, if enabled, Matrix (v3 `PUT` with txn id); an entry is marked sent if at least one channel succeeded.
Messages use Telegram HTML `parse_mode` — escape any scraped text with `html.escape`.
## Constraints
- Don't change the public API (paths, params, response shapes): `/`, `/api/v1/status`, `/update`, `/health`, `/test_message/{tg|mx}`, `/logs`. Dataclasses in `/status` are serialized as-is, so renaming their fields changes the API.
+94 -42
View File
@@ -8,12 +8,17 @@ import uvicorn
from config_file import read_yaml from config_file import read_yaml
APP_VERSION = "1.0" APP_VERSION = "1.0"
SCRIPT_PATH = os.path.dirname(__file__) SCRIPT_PATH = os.path.dirname(os.path.abspath(__file__))
CONFIGS_DIR = SCRIPT_PATH + "/configs" CONFIGS_DIR = os.path.join(SCRIPT_PATH, "configs")
CONFIG_FILE_NAME = CONFIGS_DIR + "/config.yaml" CONFIG_FILE_NAME = os.path.join(CONFIGS_DIR, "config.yaml")
TEMPLATES_DIR = os.path.join(SCRIPT_PATH, "templates")
STATIC_DIR = os.path.join(SCRIPT_PATH, "static")
TEST_DIR = os.path.join(SCRIPT_PATH, "test")
SECRETS_NAME = 'secrets.yaml' SECRETS_NAME = 'secrets.yaml'
CONFIG: Dict[str, Any] = {} CONFIG: Dict[str, Any] = {}
LOG_FORMAT = '%(asctime)s [%(levelname)s] - %(name)s: %(funcName)s[%(lineno)d] - %(message)s'
API_KEY = '' API_KEY = ''
IN_MEMORY_LOGS = False IN_MEMORY_LOGS = False
IN_MEMORY_LOGS_LEN = 100 IN_MEMORY_LOGS_LEN = 100
@@ -35,6 +40,8 @@ APARTMENT = ""
PROCESS_START_HOUR = 6 PROCESS_START_HOUR = 6
PROCESS_STOP_HOUR = 23 PROCESS_STOP_HOUR = 23
PROCESSING_MINUTES: List[int] = [5, 15, 30, 45, 55] PROCESSING_MINUTES: List[int] = [5, 15, 30, 45, 55]
HTTP_TIMEOUT = 30 # seconds, for all outgoing requests
def read_app_config(): def read_app_config():
global CONFIG global CONFIG
@@ -46,8 +53,38 @@ def read_app_config():
CONFIG = read_yaml(CONFIG_FILE_NAME, SECRETS_NAME) CONFIG = read_yaml(CONFIG_FILE_NAME, SECRETS_NAME)
def get_config_value(cfg, section, key, default):
return cfg[section][key] if section in cfg else default def get_section(cfg: Dict[str, Any], *path: str) -> Dict[str, Any]:
"""Returns nested section of config or empty dict if it is absent or not a mapping"""
section = cfg
for key in path:
section = section.get(key) if isinstance(section, dict) else None
if not isinstance(section, dict):
return {}
return section
def as_bool(value: Any) -> bool:
if isinstance(value, bool):
return value
return str(value).strip().lower() in ('true', 'yes', 'on', '1')
def as_int(value: Any, default: int, min_value: int = None, max_value: int = None) -> int:
try:
res = int(value)
except (TypeError, ValueError):
LOG.warning(f'Wrong integer value in config: {value!r}, default {default} is used')
return default
if (min_value is not None and res < min_value) or (max_value is not None and res > max_value):
LOG.warning(f'Value {res} is out of range [{min_value}, {max_value}], default {default} is used')
return default
return res
def as_str(value: Any) -> str:
return '' if value is None else str(value)
def parse_config(): def parse_config():
global API_KEY, IN_MEMORY_LOGS, IN_MEMORY_LOGS_LEN, HOST, PORT, IS_DEBUG, \ global API_KEY, IN_MEMORY_LOGS, IN_MEMORY_LOGS_LEN, HOST, PORT, IS_DEBUG, \
@@ -55,42 +92,51 @@ def parse_config():
USE_MATRIX, MX_SERVER, MX_ROOM, MX_TOKEN, \ USE_MATRIX, MX_SERVER, MX_ROOM, MX_TOKEN, \
ACCOUNT, SETTLEMENT, STREET, HOUSE, BUILDING_PART_NUMBER, APARTMENT, \ ACCOUNT, SETTLEMENT, STREET, HOUSE, BUILDING_PART_NUMBER, APARTMENT, \
PROCESS_START_HOUR, PROCESS_STOP_HOUR PROCESS_START_HOUR, PROCESS_STOP_HOUR
HOST = CONFIG.get('application', {}).get('host', '') app_section = get_section(CONFIG, 'application')
if not HOST: logs_section = get_section(app_section, 'logs_ep')
HOST = '0.0.0.0' process_section = get_section(app_section, 'process')
PORT = CONFIG.get('application', {}).get('port', 8000) tg_section = get_section(CONFIG, 'notifications', 'tg')
API_KEY = CONFIG.get('application').get('api-key', '') mx_section = get_section(CONFIG, 'notifications', 'matrix')
IN_MEMORY_LOGS = str(CONFIG.get('application', {}).get('logs_ep', {}).get('enabled', False)).lower() == 'true' data_section = get_section(CONFIG, 'data')
IN_MEMORY_LOGS_LEN = CONFIG.get('application', {}).get('logs_ep', {}).get('max_records', 100)
IS_DEBUG = CONFIG.get('application').get('debug', False) == 'true' HOST = as_str(app_section.get('host')) or '0.0.0.0'
TG_TOKEN = CONFIG.get('notifications', {}).get('tg', {}).get('token', '') PORT = as_int(app_section.get('port', 8000), 8000, 1, 65535)
TG_CHAT = CONFIG.get('notifications', {}).get('tg', {}).get('chat_id', '') API_KEY = as_str(app_section.get('api-key'))
USE_MATRIX = str(CONFIG.get('notifications', {}).get('matrix', {}).get('use_matrix', False)).lower() == 'true' IN_MEMORY_LOGS = as_bool(logs_section.get('enabled', False))
MX_SERVER = CONFIG.get('notifications', {}).get('matrix', {}).get('mx_server', '') IN_MEMORY_LOGS_LEN = as_int(logs_section.get('max_records', 100), 100, 1)
MX_ROOM = CONFIG.get('notifications', {}).get('matrix', {}).get('mx_room_id', '') IS_DEBUG = as_bool(app_section.get('debug', False))
MX_TOKEN = CONFIG.get('notifications', {}).get('matrix', {}).get('mx_access_token', '') TG_TOKEN = as_str(tg_section.get('token'))
ACCOUNT = CONFIG.get('data', {}).get('account', {}) TG_CHAT = as_str(tg_section.get('chat_id'))
SETTLEMENT = CONFIG.get('data', {}).get('settlement', {}) USE_MATRIX = as_bool(mx_section.get('use_matrix', False))
STREET = CONFIG.get('data', {}).get('street', {}) MX_SERVER = as_str(mx_section.get('mx_server')).rstrip('/')
HOUSE = CONFIG.get('data', {}).get('house', {}) MX_ROOM = as_str(mx_section.get('mx_room_id'))
BUILDING_PART_NUMBER = CONFIG.get('data', {}).get('building_part_number', {}) MX_TOKEN = as_str(mx_section.get('mx_access_token'))
APARTMENT = CONFIG.get('data', {}).get('apartment', {}) ACCOUNT = as_str(data_section.get('account'))
PROCESS_START_HOUR = CONFIG.get('application').get('process', {}).get('start_hour', 6) SETTLEMENT = as_str(data_section.get('settlement'))
PROCESS_STOP_HOUR = CONFIG.get('application').get('process', {}).get('stop_hour', 23) STREET = as_str(data_section.get('street'))
if PROCESS_START_HOUR > PROCESS_STOP_HOUR: HOUSE = as_str(data_section.get('house'))
PROCESS_STOP_HOUR = 23 BUILDING_PART_NUMBER = as_str(data_section.get('building_part_number'))
APARTMENT = as_str(data_section.get('apartment'))
PROCESS_START_HOUR = as_int(process_section.get('start_hour', 6), 6, 0, 23)
PROCESS_STOP_HOUR = as_int(process_section.get('stop_hour', 23), 23, 1, 24)
if PROCESS_START_HOUR >= PROCESS_STOP_HOUR:
LOG.warning(f'start_hour {PROCESS_START_HOUR} >= stop_hour {PROCESS_STOP_HOUR}, stop_hour 24 is used')
PROCESS_STOP_HOUR = 24
if not TG_TOKEN or not TG_CHAT:
LOG.warning('Telegram token or chat_id is not configured')
if USE_MATRIX and not (MX_SERVER and MX_ROOM and MX_TOKEN):
LOG.warning('Matrix is enabled but not fully configured')
def get_uvicorn_config(): def get_uvicorn_config():
# global HOST, PORT
if HOST and PORT: if HOST and PORT:
cfg = { cfg = {
"host": f"{HOST}", "host": f"{HOST}",
"port": PORT, "port": PORT,
"access_log": True, "access_log": True,
"log_level": "info", "log_level": "info",
"log_config": f"{CONFIGS_DIR}/log.yaml" "log_config": os.path.join(CONFIGS_DIR, "log.yaml")
# "docs_url": "/api-docs"
} }
else: else:
cfg = { cfg = {
@@ -99,8 +145,7 @@ def get_uvicorn_config():
"reload": True, "reload": True,
"access_log": False, "access_log": False,
"log_level": "info", "log_level": "info",
"log_config": f"{CONFIGS_DIR}/log.yaml" "log_config": os.path.join(CONFIGS_DIR, "log.yaml")
# "docs_url": "/api-docs"
} }
LOG.debug(f'got uvicorn_config: {cfg}') LOG.debug(f'got uvicorn_config: {cfg}')
@@ -109,25 +154,32 @@ def get_uvicorn_config():
def log_config(): # Not used due to not working def log_config(): # Not used due to not working
cfg = uvicorn.config.LOGGING_CONFIG cfg = uvicorn.config.LOGGING_CONFIG
cfg["formatters"]["access"]["fmt"] = ( cfg["formatters"]["access"]["fmt"] = LOG_FORMAT
'%(asctime)s [%(levelname)s] - %(name)s: %(funcName)s[%(lineno)d] - %(message)s'
)
return cfg return cfg
LOG_LIST = []
LOG_LIST: List[str] = []
class ListHandler(logging.Handler): class ListHandler(logging.Handler):
def emit(self, record): def emit(self, record):
if IN_MEMORY_LOGS: if not IN_MEMORY_LOGS:
return
try:
LOG_LIST.append(self.format(record)) LOG_LIST.append(self.format(record))
while len(LOG_LIST) > IN_MEMORY_LOGS_LEN: if len(LOG_LIST) > IN_MEMORY_LOGS_LEN:
LOG_LIST.pop(0) del LOG_LIST[:len(LOG_LIST) - IN_MEMORY_LOGS_LEN]
except Exception:
self.handleError(record)
### ###
LOG = logging.getLogger(__name__) LOG = logging.getLogger(__name__)
handler = ListHandler() handler = ListHandler()
handler.setFormatter(logging.Formatter('%(asctime)s [%(levelname)s] - %(name)s: %(funcName)s[%(lineno)d] - %(message)s')) handler.setFormatter(logging.Formatter(LOG_FORMAT))
LOG.addHandler(handler) LOG.addHandler(handler)
def init_config(): def init_config():
read_app_config() read_app_config()
parse_config() parse_config()
+29 -33
View File
@@ -1,43 +1,43 @@
import json import json
import os import os
import yaml import yaml
def read_config(name): def read_config(name):
if not os.path.isfile(name): if not os.path.isfile(name):
raise Exception(f"File {name} doesn't exists") raise FileNotFoundError(f"File {name} doesn't exist")
filename, ext = os.path.splitext(name) _, ext = os.path.splitext(name)
if 'json' in ext: ext = ext.lower()
if ext == '.json':
return read_json(name) return read_json(name)
elif 'properties' in ext: elif ext == '.properties':
return read_prop(name) return read_prop(name)
elif 'yaml' in ext or 'yml' in ext: elif ext in ('.yaml', '.yml'):
return read_yaml(name) return read_yaml(name)
else: else:
raise Exception("Wrong file type") raise ValueError(f"Wrong file type: {ext}")
def read_json(name): def read_json(name):
with open(name, 'r', encoding='utf-8') as f: with open(name, 'r', encoding='utf-8') as f:
j_conf = json.load(f) return dict(json.load(f))
conf = {}
for key, value in j_conf.items():
conf[key] = value
return conf
def read_prop(filepath, sep='=', comment_char='#'): def read_prop(filepath, sep='=', comment_char='#'):
""" """
Read the file passed as parameter as a properties file. Read the file passed as parameter as a properties file.
""" """
conf = {} conf = {}
with open(filepath, "rt") as f: with open(filepath, "rt", encoding='utf-8') as f:
for line in f: for line in f:
l = line.strip() l = line.strip()
if l and not l.startswith(comment_char): if l and not l.startswith(comment_char):
key_value = l.split(sep) key, _, value = l.partition(sep)
key = key_value[0].strip() conf[key.strip()] = value.strip().strip('"')
value = sep.join(key_value[1:]).strip().strip('"')
conf[key] = value
return conf return conf
def read_yaml(name, secrets_file='secrets.yaml'): def read_yaml(name, secrets_file='secrets.yaml'):
# Load secrets first # Load secrets first
secrets = {} secrets = {}
@@ -47,28 +47,24 @@ def read_yaml(name, secrets_file='secrets.yaml'):
with open(secrets_path, 'r', encoding='utf-8') as f: with open(secrets_path, 'r', encoding='utf-8') as f:
secrets = yaml.safe_load(f) or {} secrets = yaml.safe_load(f) or {}
# Define a custom constructor for !secret tag
def secret_constructor(loader, node): def secret_constructor(loader, node):
secret_key = loader.construct_scalar(node) secret_key = loader.construct_scalar(node)
if secret_key not in secrets: if secret_key not in secrets:
raise ValueError(f"Secret '{secret_key}' not found in {secrets_file}") raise ValueError(f"Secret '{secret_key}' not found in {secrets_file}")
return secrets[secret_key] return secrets[secret_key]
# Register the custom constructor # Own loader class, so the global yaml.SafeLoader is not modified
yaml.add_constructor('!secret', secret_constructor, Loader=yaml.SafeLoader) class SecretLoader(yaml.SafeLoader):
# Load the main configuration
conf = {}
with open(name, 'r', encoding='utf-8') as f:
y_conf = yaml.safe_load(f)
if y_conf:
for key, value in y_conf.items():
conf[key] = value
return conf
def main():
pass pass
if __name__ == "__main__": SecretLoader.add_constructor('!secret', secret_constructor)
main()
# Load the main configuration
with open(name, 'r', encoding='utf-8') as f:
y_conf = yaml.load(f, Loader=SecretLoader)
if y_conf is None:
return {}
if not isinstance(y_conf, dict):
raise ValueError(f"Wrong config structure in {name}: top level must be a mapping")
return dict(y_conf)
+73 -54
View File
@@ -1,16 +1,19 @@
import asyncio
import logging import logging
import os
from contextlib import asynccontextmanager, suppress
from datetime import datetime, UTC from datetime import datetime, UTC
from typing import Optional, List from typing import Optional, List
import uvicorn import uvicorn
from apscheduler.schedulers.asyncio import AsyncIOScheduler from apscheduler.schedulers.asyncio import AsyncIOScheduler
from fastapi import FastAPI, BackgroundTasks, Request from fastapi import FastAPI, Request
from starlette.responses import FileResponse, HTMLResponse from starlette.responses import FileResponse, HTMLResponse
from starlette.routing import Mount from starlette.routing import Mount
from starlette.templating import Jinja2Templates from starlette.templating import Jinja2Templates
import app_config as cfg import app_config as cfg
from model import POData, JobData from model import JobData
from utils import process_gpv, process_po, get_parsed_po_data, get_parsed_gpv_data, send_message_to_tg_get, \ from utils import process_gpv, process_po, get_parsed_po_data, get_parsed_gpv_data, send_message_to_tg_get, \
send_matrix_notification send_matrix_notification
@@ -26,38 +29,33 @@ Created by -=:dAs:=-
''' '''
######################## ########################
# cfg.init_config() ToDo: TEST IT! cfg.init_config()
SCHEDULER: Optional[AsyncIOScheduler] = None SCHEDULER: Optional[AsyncIOScheduler] = None
LOG = cfg.LOG LOG = cfg.LOG
TEMPLATES = Jinja2Templates(directory="templates") TEMPLATES = Jinja2Templates(directory=cfg.TEMPLATES_DIR)
######################## # prevents simultaneous processing (scheduled job and /update) and duplicated notifications
TAGS_METADATA = [] PROCESS_LOCK = asyncio.Lock()
app = FastAPI( INITIAL_CHECK_TASK: Optional[asyncio.Task] = None
title="Energy Outage Inform Service by -=:dAs:=-",
description=DESCRIPTION,
version=cfg.APP_VERSION,
license_info={
"name": "Apache 2.0",
"url": "https://www.apache.org/licenses/LICENSE-2.0.html",
},
openapi_tags=TAGS_METADATA
)
api = FastAPI()
app.mount("/api/v1", api)
######################## ########################
async def check_and_notify(): async def check_and_notify():
await process_gpv() async with PROCESS_LOCK:
await process_po() for process in (process_gpv, process_po):
try:
await process()
except Exception:
LOG.exception(f'Unexpected error in {process.__name__}')
@app.on_event("startup") @asynccontextmanager
async def startup_event(): async def lifespan(_: FastAPI):
global SCHEDULER global SCHEDULER, INITIAL_CHECK_TASK
# --- STARTUP LOGIC --- # --- STARTUP LOGIC ---
LOG.info("--- Starting EO Notifier...") LOG.info("--- Starting EO Notifier...")
SCHEDULER = AsyncIOScheduler() SCHEDULER = AsyncIOScheduler(job_defaults={'coalesce': True, 'max_instances': 1, 'misfire_grace_time': 60})
mins = cfg.PROCESSING_MINUTES if not cfg.IS_DEBUG else list(range(0, 60, 2))
for h in range(cfg.PROCESS_START_HOUR, cfg.PROCESS_STOP_HOUR): for h in range(cfg.PROCESS_START_HOUR, cfg.PROCESS_STOP_HOUR):
mins = cfg.PROCESSING_MINUTES if not cfg.IS_DEBUG else [x for x in range(0, 60) if x % 2 == 0]
for m in mins: for m in mins:
SCHEDULER.add_job( SCHEDULER.add_job(
check_and_notify, check_and_notify,
@@ -71,27 +69,54 @@ async def startup_event():
SCHEDULER.start() SCHEDULER.start()
LOG.info("Scheduler started") LOG.info("Scheduler started")
# Run initial check # Run initial check in background, so service starts even if data sources are unavailable
await check_and_notify() INITIAL_CHECK_TASK = asyncio.create_task(check_and_notify())
yield
@app.on_event("shutdown")
async def shutdown_event():
# --- SHUTDOWN LOGIC --- # --- SHUTDOWN LOGIC ---
if SCHEDULER: if INITIAL_CHECK_TASK and not INITIAL_CHECK_TASK.done():
SCHEDULER.shutdown() INITIAL_CHECK_TASK.cancel()
with suppress(asyncio.CancelledError):
await INITIAL_CHECK_TASK
if SCHEDULER and SCHEDULER.running:
SCHEDULER.shutdown(wait=False)
LOG.info('Scheduler stopped') LOG.info('Scheduler stopped')
LOG.info('--- EO Notifier shutting down') LOG.info('--- EO Notifier shutting down')
TAGS_METADATA = []
app = FastAPI(
title="Energy Outage Inform Service by -=:dAs:=-",
description=DESCRIPTION,
version=cfg.APP_VERSION,
license_info={
"name": "Apache 2.0",
"url": "https://www.apache.org/licenses/LICENSE-2.0.html",
},
openapi_tags=TAGS_METADATA,
lifespan=lifespan
)
api = FastAPI()
app.mount("/api/v1", api)
########################
def is_scheduler_running() -> bool:
return bool(SCHEDULER and SCHEDULER.running)
@app.get("/favicon.ico", include_in_schema=False) @app.get("/favicon.ico", include_in_schema=False)
async def get_favicon(): async def get_favicon():
return FileResponse('static/favicon.png') return FileResponse(os.path.join(cfg.STATIC_DIR, 'favicon.png'))
def list_endpoints(): def list_endpoints():
url_list = [r for r in api.routes if 'HEAD' not in r.methods] url_list = [r for r in api.routes if 'HEAD' not in (getattr(r, 'methods', None) or {'HEAD'})]
return url_list return url_list
@@ -108,12 +133,11 @@ async def root(request: Request):
"Service": "Energy Outage Inform Service by -=:dAs:=", "Service": "Energy Outage Inform Service by -=:dAs:=",
"version": f"{cfg.APP_VERSION}", "version": f"{cfg.APP_VERSION}",
"status": "running", "status": "running",
"Scheduler running": SCHEDULER.running, "Scheduler running": is_scheduler_running(),
"timestamp": date_format(datetime.now(UTC)) "timestamp": date_format(datetime.now(UTC))
} }
return TEMPLATES.TemplateResponse("info.html", { return TEMPLATES.TemplateResponse(request, "info.html", {
"request": request,
"help_messages": help_messages, "help_messages": help_messages,
"title": "Energy Outage Inform Service by -=:dAs:=", "title": "Energy Outage Inform Service by -=:dAs:=",
"EPs": list_endpoints(), "EPs": list_endpoints(),
@@ -128,13 +152,12 @@ async def root(request: Request):
description='Showing current status of the service') description='Showing current status of the service')
async def get_status(): async def get_status():
"""Detailed health check""" """Detailed health check"""
global SCHEDULER
return { return {
"status": "healthy", "status": "healthy",
"config_loaded": bool(cfg.CONFIG), "config_loaded": bool(cfg.CONFIG),
"PO_data": get_parsed_po_data(), "PO_data": get_parsed_po_data(),
"GPV_data": get_parsed_gpv_data(), "GPV_data": get_parsed_gpv_data(),
"scheduler_running": SCHEDULER.running, "scheduler_running": is_scheduler_running(),
"scheduler_jobs": await get_scheduled_jobs() "scheduler_jobs": await get_scheduled_jobs()
} }
@@ -149,17 +172,9 @@ async def update():
async def get_scheduled_jobs() -> List[JobData]: async def get_scheduled_jobs() -> List[JobData]:
global SCHEDULER if not SCHEDULER:
jobs: List[JobData] = [] return []
for j in SCHEDULER.get_jobs(): return [JobData(name=j.id, trigger=str(j.trigger), next=j.next_run_time) for j in SCHEDULER.get_jobs()]
jobs.append(
JobData(
name=j.id,
trigger=str(j.trigger),
next=j.next_run_time)
)
return jobs
@api.get("/health", @api.get("/health",
@@ -174,17 +189,21 @@ async def health_check():
@api.get("/test_message/{messanger}", @api.get("/test_message/{messanger}",
summary='Send message to messanger', summary='Send message to messanger',
description='Send test message to messanger according to current configuration') description='Send test message to messanger according to current configuration')
async def send_test_message(messanger: str = None): async def send_test_message(messanger: str):
messanger_up = messanger.upper() messanger_up = messanger.upper()
msg = f'Test message to {messanger_up}' msg = f'Test message to {messanger_up}'
if messanger_up == "TG": if messanger_up == "TG":
send_message_to_tg_get(msg) is_sent = await asyncio.to_thread(send_message_to_tg_get, msg) == 200
elif messanger_up == "MX": elif messanger_up == "MX":
send_matrix_notification(msg) is_sent = await asyncio.to_thread(send_matrix_notification, msg)
else: else:
return { return {
"status": f"Error: Wrong messanger '{messanger_up}'" "status": f"Error: Wrong messanger '{messanger_up}'"
} }
if not is_sent:
return {
"status": f"Error: message to '{messanger_up}' was not sent"
}
return { return {
"status": "OK" "status": "OK"
} }
@@ -194,7 +213,7 @@ async def send_test_message(messanger: str = None):
summary='Show LOGs', summary='Show LOGs',
description='Showing Service''s LOGs if it configured') description='Showing Service''s LOGs if it configured')
async def get_logs(): async def get_logs():
return {'logs': cfg.LOG_LIST} return {'logs': list(cfg.LOG_LIST)}
def date_format(dt: datetime, fmt: str = '%Y-%m-%d %H:%M:%S'): def date_format(dt: datetime, fmt: str = '%Y-%m-%d %H:%M:%S'):
@@ -204,7 +223,7 @@ def date_format(dt: datetime, fmt: str = '%Y-%m-%d %H:%M:%S'):
def main(): def main():
logging.basicConfig( logging.basicConfig(
level=logging.INFO, level=logging.INFO,
format='%(asctime)s [%(levelname)s] - %(name)s: %(funcName)s[%(lineno)d] - %(message)s' format=cfg.LOG_FORMAT
) )
cfg.LOG.info('Application starting') cfg.LOG.info('Application starting')
uvicorn.run("main:app", **cfg.get_uvicorn_config()) uvicorn.run("main:app", **cfg.get_uvicorn_config())
+67 -20
View File
@@ -1,15 +1,28 @@
from dataclasses import dataclass, field from dataclasses import dataclass, field
from datetime import datetime from datetime import datetime, timedelta
from typing import List, Self from typing import List, Optional, Self
import app_config as cfg import app_config as cfg
LOG = cfg.LOG LOG = cfg.LOG
PO_DATETIME_FORMAT = '%Y.%m.%d %H:%M'
GPV_DATETIME_FORMAT = '%d.%m.%Y %H:%M'
GPV_DATE_FORMAT = '%d.%m.%Y'
def parse_ts(value: str, fmt: str) -> Optional[float]:
try:
return datetime.strptime(value, fmt).timestamp()
except (TypeError, ValueError):
return None
@dataclass @dataclass
class POData: class POData:
city: str = field(repr=False) city: str = field(repr=False)
street: str = field(repr=False) street: str = field(repr=False)
house: int = field(repr=False) house: str = field(repr=False)
po_type: str = field(repr=False) po_type: str = field(repr=False)
reason: str = field(repr=False) reason: str = field(repr=False)
placement_date: str placement_date: str
@@ -17,16 +30,23 @@ class POData:
stop_date: str stop_date: str
start_time: str start_time: str
stop_time: str stop_time: str
notified_at: datetime = field(repr=False, default=None) # service fields are not compared: equality means the same outage content
updated_at: datetime = field(repr=False, default=datetime.now()) notified_at: Optional[datetime] = field(repr=False, default=None, compare=False)
need_to_send: bool = field(repr=False, default=True) updated_at: datetime = field(repr=False, default_factory=datetime.now, compare=False)
start_ts: float = field(init=False) need_to_send: bool = field(repr=False, default=True, compare=False)
start_ts: float = field(init=False, default=0.0, compare=False)
def __post_init__(self): def __post_init__(self):
if self.start_date and self.start_time: ts = parse_ts(f'{self.start_date} {self.start_time}', PO_DATETIME_FORMAT)
self.start_ts = datetime.strptime(f'{self.start_date} {self.start_time}', '%Y.%m.%d %H:%M').timestamp() if ts is None:
else:
LOG.error(f'wrong start_date or start_time in {self}') LOG.error(f'wrong start_date or start_time in {self}')
else:
self.start_ts = ts
@property
def key(self):
"""Identity of the outage: the same address and start time"""
return self.start_ts, self.city, self.street, self.house
def set_sent(self): def set_sent(self):
self.need_to_send = False self.need_to_send = False
@@ -45,17 +65,44 @@ class GPVData:
hours_off: List hours_off: List
hours_on: List hours_on: List
outage_queue: str = field(repr=False) outage_queue: str = field(repr=False)
notified_at: datetime = field(repr=False, default=None) notified_at: Optional[datetime] = field(repr=False, default=None, compare=False)
updated_at: datetime = field(repr=False, default=datetime.now()) updated_at: datetime = field(repr=False, default_factory=datetime.now, compare=False)
need_to_send: bool = field(repr=False, default=True) need_to_send: bool = field(repr=False, default=True, compare=False)
start_ts: List[float] = field(default_factory=list, init=False) # timestamps of power-on moments (ends of outages)
start_ts: List[float] = field(default_factory=list, init=False, compare=False)
def __post_init__(self): def __post_init__(self):
if self.event_date and self.hours_off and self.hours_on: day_start = parse_ts(f'{self.event_date} 00:00', GPV_DATETIME_FORMAT)
for index, value in enumerate(self.hours_on): if day_start is None:
self.start_ts.append(datetime.strptime(f'{self.event_date} {self.hours_on[index]}', '%d.%m.%Y %H:%M').timestamp()) LOG.error(f'wrong event_date in {self}')
else: return
LOG.error(f'wrong start_date or start_time in {self}') for hour_off, hour_on in zip(self.hours_off, self.hours_on):
ts_off = parse_ts(f'{self.event_date} {hour_off}', GPV_DATETIME_FORMAT)
ts_on = parse_ts(f'{self.event_date} {hour_on}', GPV_DATETIME_FORMAT)
if ts_on is None:
LOG.error(f'wrong time {hour_on!r} in {self}')
continue
if ts_off is not None and ts_on <= ts_off:
# e.g. '22:00-00:00': power is on at the next day
ts_on = (datetime.fromtimestamp(ts_on) + timedelta(days=1)).timestamp()
self.start_ts.append(ts_on)
@property
def key(self):
return self.event_date, self.outage_queue
@property
def actual_till_ts(self) -> Optional[float]:
"""Moment after which the schedule is not actual: the last power-on or end of the event day"""
if self.start_ts:
return max(self.start_ts)
day_start = parse_ts(f'{self.event_date} 00:00', GPV_DATETIME_FORMAT)
if day_start is None:
return None
return (datetime.fromtimestamp(day_start) + timedelta(days=1)).timestamp()
def is_same_schedule(self, other: Self) -> bool:
return self.hours_off == other.hours_off and self.hours_on == other.hours_on
def set_send(self): def set_send(self):
self.need_to_send = False self.need_to_send = False
@@ -66,7 +113,7 @@ class GPVData:
class JobData: class JobData:
name: str name: str
trigger: str trigger: str
next: str next: Optional[datetime]
if __name__ == '__main__': if __name__ == '__main__':
+66
View File
@@ -0,0 +1,66 @@
{
"current": {
"gav": {
"message": "Черга спеціальних аварійних відключень (СГАВ):",
"queue": null
},
"hasQueue": "yes",
"note": "Станом на 21:05 14.01.2026 за вказаним особовим рахунком '30014180' споживач підпадає під чергу '5.1' Графіку погодинного відключення(ГПВ)",
"queue": 5,
"sgav": {
"message": "Черга спеціальних аварійних відключень (СГАВ):",
"queue": null
},
"subQueue": 1
},
"schedule": [
{
"createdAt": "14.01.2026 19:11",
"eventDate": "15.04.2026",
"queues": {
"5.1": [
{
"from": "07:30",
"shutdownHours": "07:30-11:00",
"status": 1,
"to": "11:00"
},
{
"from": "14:30",
"shutdownHours": "14:30-20:00",
"status": 1,
"to": "20:00"
},
{
"from": "22:00",
"shutdownHours": "22:00-00:00",
"status": 1,
"to": "00:00"
}
]
},
"scheduleApprovedSince": "14.04.2026 19:11"
},
{
"createdAt": "13.04.2026 20:28",
"eventDate": "14.04.2026",
"queues": {
"5.1": [
{
"from": "03:00",
"shutdownHours": "03:00-07:00",
"status": 1,
"to": "07:00"
},
{
"from": "11:00",
"shutdownHours": "11:00-17:30",
"status": 1,
"to": "17:30"
}
]
},
"scheduleApprovedSince": "14.04.2026 09:56"
}
]
}
+289 -282
View File
@@ -1,15 +1,19 @@
import asyncio
import html
import json import json
import os import os
import re
import uuid
from datetime import datetime, timedelta from datetime import datetime, timedelta
from io import StringIO from typing import List, Optional
from typing import List from urllib.parse import quote
import requests import requests
import urllib3 import urllib3
from lxml import etree from lxml import etree
import app_config as cfg import app_config as cfg
from model import POData, GPVData from model import POData, GPVData, GPV_DATE_FORMAT
STATUS_SEND_OK = 0 STATUS_SEND_OK = 0
STATUS_ERROR_SEND_TO_TG = 20 STATUS_ERROR_SEND_TO_TG = 20
@@ -23,24 +27,59 @@ PARSED_GPV_DATAS: List[GPVData] = []
LOG = cfg.LOG LOG = cfg.LOG
def send_message_to_tg_get(msg: str): # be-svitlo.oe.if.ua is requested without certificate verification
url = f"https://api.telegram.org/bot{cfg.TG_TOKEN}/sendMessage?chat_id={cfg.TG_CHAT}&parse_mode=html&text={msg}" urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
if not cfg.IS_DEBUG:
return requests.get(url).status_code DIGIT_CHARS = [f'{d}\U000020E3' for d in range(10)]
else:
def safe_error(e: Exception) -> str:
"""Error description without secrets (request URLs may contain the TG token)"""
msg = f'{type(e).__name__}: {e}'
for secret in (cfg.TG_TOKEN, cfg.MX_TOKEN):
if secret:
msg = msg.replace(secret, '***')
return msg
def send_message_to_tg_get(msg: str) -> int:
"""Returns HTTP status code of Telegram response or 0 if the request failed"""
if cfg.IS_DEBUG:
print(f"Send to TG GET: {msg}") print(f"Send to TG GET: {msg}")
return 200 return 200
if not cfg.TG_TOKEN or not cfg.TG_CHAT:
LOG.error('Telegram is not configured')
return 0
try:
response = requests.get(
f"https://api.telegram.org/bot{cfg.TG_TOKEN}/sendMessage",
params={'chat_id': cfg.TG_CHAT, 'parse_mode': 'html', 'text': msg},
timeout=cfg.HTTP_TIMEOUT)
except requests.RequestException as e:
LOG.error(f'Error sending TG message: {safe_error(e)}')
return 0
if response.status_code != 200:
LOG.error(f'TG response code={response.status_code}: {response.text[:500]}')
return response.status_code
def send_message_to_tg_post(msg: str): def send_message_to_tg_post(msg: str) -> int:
if not cfg.IS_DEBUG: """Returns HTTP status code of Telegram response or 0 if the request failed"""
return requests.post( if cfg.IS_DEBUG:
url='https://api.telegram.org/bot{0}/{1}'.format(cfg.TG_TOKEN, 'sendMessage'),
data={'chat_id': {cfg.TG_CHAT}, 'text': {msg}, 'parse_mode': 'html'}
).status_code
else:
print(f"Send to TG POST: {msg}") print(f"Send to TG POST: {msg}")
return 200 return 200
if not cfg.TG_TOKEN or not cfg.TG_CHAT:
LOG.error('Telegram is not configured')
return 0
try:
return requests.post(
url=f'https://api.telegram.org/bot{cfg.TG_TOKEN}/sendMessage',
data={'chat_id': cfg.TG_CHAT, 'text': msg, 'parse_mode': 'html'},
timeout=cfg.HTTP_TIMEOUT
).status_code
except requests.RequestException as e:
LOG.error(f'Error sending TG message: {safe_error(e)}')
return 0
def send_matrix_notification(message: str) -> bool: def send_matrix_notification(message: str) -> bool:
@@ -48,30 +87,51 @@ def send_matrix_notification(message: str) -> bool:
if not cfg.USE_MATRIX: if not cfg.USE_MATRIX:
return False return False
if not cfg.IS_DEBUG: if cfg.IS_DEBUG:
try: print(f"Send to MX: {message}")
url = f"{cfg.MX_SERVER}/_matrix/client/r0/rooms/{cfg.MX_ROOM}/send/m.room.message" return True
if not (cfg.MX_SERVER and cfg.MX_ROOM and cfg.MX_TOKEN):
LOG.error('Matrix is not configured')
return False
url = (f"{cfg.MX_SERVER}/_matrix/client/v3/rooms/{quote(cfg.MX_ROOM, safe='')}"
f"/send/m.room.message/{uuid.uuid4().hex}")
headers = { headers = {
'Authorization': f'Bearer {cfg.MX_TOKEN}', 'Authorization': f'Bearer {cfg.MX_TOKEN}',
'Content-Type': 'application/json' 'Content-Type': 'application/json'
} }
payload = { payload = {
'msgtype': 'm.text', 'msgtype': 'm.text',
'body': message, 'body': html.unescape(re.sub(r'<[^>]+>', '', message)),
'format': 'org.matrix.custom.html', 'format': 'org.matrix.custom.html',
'formatted_body': message.replace('\n', '<br/>') 'formatted_body': message.replace('\n', '<br/>')
} }
try:
response = requests.post(url, headers=headers, json=payload) response = requests.put(url, headers=headers, json=payload, timeout=cfg.HTTP_TIMEOUT)
return response.status_code == 200 except requests.RequestException as e:
except Exception as e: LOG.error(f"Error sending Matrix notification: {safe_error(e)}")
# logger.error(f"Error sending Matrix notification: {e}")
return False return False
if response.status_code != 200:
LOG.error(f'MX response code={response.status_code}: {response.text[:500]}')
return response.status_code == 200
def notify(message: str) -> bool:
"""Sends message to all configured messengers. Returns True if it was delivered at least to one of them"""
delivered = False
if send_message_to_tg_get(message) == 200:
delivered = True
LOG.debug(f'TG message send {message}')
else: else:
print(f"Send to MX: {message}") LOG.error(f'TG send error code: {STATUS_ERROR_SEND_TO_TG}')
return True
if cfg.USE_MATRIX:
if send_matrix_notification(message):
delivered = True
else:
LOG.error(f'MX send error code: {STATUS_ERROR_SEND_TO_MX}')
return delivered
def get_GPV_response_from_oe(account): def get_GPV_response_from_oe(account):
@@ -95,8 +155,7 @@ def get_GPV_response_from_oe(account):
"userSearchChoice": "pob", "userSearchChoice": "pob",
"address": "" "address": ""
} }
urllib3.disable_warnings() return requests.post(url, headers=headers, data=data, verify=False, timeout=cfg.HTTP_TIMEOUT)
return requests.post(url, headers=headers, data=data, verify=False)
def get_PO_response(): # Power Outage def get_PO_response(): # Power Outage
@@ -115,8 +174,7 @@ def get_PO_response(): # Power Outage
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36' 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36'
} }
urllib3.disable_warnings() return requests.get(url=url, headers=headers, params=params, timeout=cfg.HTTP_TIMEOUT)
return requests.get(url=url, headers=headers, params=params)
def has_numbers(in_str): def has_numbers(in_str):
@@ -124,55 +182,27 @@ def has_numbers(in_str):
def get_utf_chars(number, is_time=False): def get_utf_chars(number, is_time=False):
digit_chars = ['\U00000030\U000020E3', '\U00000031\U000020E3', '\U00000032\U000020E3', '\U00000033\U000020E3', """Replaces ASCII digits with keycap emoji digits. is_time is kept for compatibility"""
'\U00000034\U000020E3', '\U00000035\U000020E3', '\U00000036\U000020E3', '\U00000037\U000020E3', return ''.join(DIGIT_CHARS[int(c)] if c in '0123456789' else c for c in str(number))
'\U00000038\U000020E3', '\U00000039\U000020E3']
res = ''
if is_time:
h = number.split(':')[0]
m = number.split(':')[1]
if m != 0:
value = str(h) + ':' + str(m)
else:
value = str(h) + ':00'
else:
value = str(number)
for c in value:
if c.isdigit and c not in ['.', ':']:
res += digit_chars[int(c)]
else:
res += c
return res
def get_digit_message(arr, is_time=False): def get_digit_message(arr, is_time=False):
first = 0 first = arr[0] if len(arr) > 0 else 0
if len(arr) > 0: last = arr[1] if len(arr) > 1 else 0
first = arr[0] return f'з {get_utf_chars(first, is_time)} до {get_utf_chars(last, is_time)}'
last = 0
if len(arr) > 1:
last = arr[1]
msg = 'з {num0} до {num1}'
return msg.format(num0=get_utf_chars(first, is_time), num1=get_utf_chars(last, is_time))
def get_GPV_message(approved_from, event_date, hours_off, hours_on, outage_queue): def get_GPV_message(approved_from, event_date, hours_off, hours_on, outage_queue):
queue = get_utf_chars(outage_queue) queue = get_utf_chars(html.escape(str(outage_queue)))
message = f'\U000026A1 <b>ГПВ на {event_date} для {queue} черги</b>' message = f'\U000026A1 <b>ГПВ на {html.escape(str(event_date))} для {queue} черги</b>'
if len(hours_off) > 0: if len(hours_off) > 0:
message += '\nВимкнення:' message += '\nВимкнення:'
for i in range(len(hours_off)): for hour_off, hour_on in zip(hours_off, hours_on):
arr = [hours_off[i], hours_on[i]] message += '\n' + get_digit_message([html.escape(str(hour_off)), html.escape(str(hour_on))], True)
msg = get_digit_message(arr, True)
message += '\n' + msg
# message += (f'\n{MENTIONS}')
else: else:
message += '\nВимкнень немає \U0001F44C \U0001F483' message += '\nВимкнень немає \U0001F44C \U0001F483'
message += (f'\n<b><i>{approved_from}</i></b>') message += f'\n<b><i>{html.escape(str(approved_from))}</i></b>'
return message return message
@@ -181,44 +211,57 @@ def get_address(city, street, house):
def is_outage_date_before_or_same(stop_date): def is_outage_date_before_or_same(stop_date):
"""True if the outage is not finished before today (or the date can't be parsed)"""
try:
parsed_date = datetime.strptime(stop_date, "%Y.%m.%d").date() parsed_date = datetime.strptime(stop_date, "%Y.%m.%d").date()
current_date = datetime.now().date() except (TypeError, ValueError):
return parsed_date + timedelta(hours=1) >= current_date LOG.warning(f'Wrong stop date: {stop_date!r}')
return True
return parsed_date >= datetime.now().date()
def get_row_text(row, path: str) -> str:
return ' '.join(t.strip() for t in row.xpath(path) if t and t.strip())
def parse_po_data(root) -> List[POData]: def parse_po_data(root) -> List[POData]:
res: List[POData] = [] res: List[POData] = []
for i in range(len(root.findall(".//div[@class='table-row flex']"))): for row in root.xpath("//div[@class='table-row flex']"):
po_data = POData( po_data = POData(
city=root.xpath("//div[@class='table-row flex']/div[@class='city']/text()")[i], city=get_row_text(row, "./div[@class='city']//text()"),
street=root.xpath("//div[@class='table-row flex']/div[@class='street']/text()")[i], street=get_row_text(row, "./div[@class='street']//text()"),
house=root.xpath("//div[@class='table-row flex']/div[@class='house_number']/text()")[i], house=get_row_text(row, "./div[@class='house_number']//text()"),
po_type=root.xpath("//div[@class='table-row flex']/div[@class='type-shutdown']/text()")[i], po_type=get_row_text(row, "./div[@class='type-shutdown']//text()"),
reason=root.xpath("//div[@class='table-row flex']/div[@class='reason-shutdown reason-text-container']/div[@class='reason-text-hide']/text()")[i], reason=get_row_text(row, "./div[@class='reason-shutdown reason-text-container']/div[@class='reason-text-hide']//text()"),
placement_date=root.xpath("//div[@class='table-row flex']/div[@class='placement_date']/div[@class='date']/text()")[i], placement_date=get_row_text(row, "./div[@class='placement_date']/div[@class='date']//text()"),
start_date=root.xpath("//div[@class='table-row flex']/div[@class='shutdown_date']/div[@class='date']/text()")[i], start_date=get_row_text(row, "./div[@class='shutdown_date']/div[@class='date']//text()"),
stop_date=root.xpath("//div[@class='table-row flex']/div[@class='turn_on_date']/div[@class='date']/text()")[i], stop_date=get_row_text(row, "./div[@class='turn_on_date']/div[@class='date']//text()"),
start_time=root.xpath("//div[@class='table-row flex']/div[@class='shutdown_date']/div[@class='time']/text()")[i], start_time=get_row_text(row, "./div[@class='shutdown_date']/div[@class='time']//text()"),
stop_time=root.xpath("//div[@class='table-row flex']/div[@class='turn_on_date']/div[@class='time']/text()")[i]) stop_time=get_row_text(row, "./div[@class='turn_on_date']/div[@class='time']//text()"))
if is_outage_date_before_or_same(po_data.stop_date): if not po_data.start_ts:
po_data.need_to_send = True LOG.warning(f'PO row skipped due to wrong start date/time: {po_data}')
continue
if not is_outage_date_before_or_same(po_data.stop_date):
continue
if po_data not in res: if po_data not in res:
res.append(po_data) res.append(po_data)
res.sort(key=lambda pd: pd.start_date) res.sort(key=lambda pd: pd.start_ts)
return res return res
def load_last_datas(file_name): def load_last_datas(file_name):
if os.path.isfile(file_name): if os.path.isfile(file_name):
with open(file_name, 'r') as f: with open(file_name, 'r', encoding='utf-8') as f:
return f.read() return f.read()
else: else:
return '' return ''
def get_parsed_po_data(): def get_parsed_po_data():
return PARSED_PO_DATAS return PARSED_PO_DATAS
@@ -227,118 +270,50 @@ def get_parsed_gpv_data():
return PARSED_GPV_DATAS return PARSED_GPV_DATAS
async def process_gpv(): def gpv_date_key(entry: dict):
if cfg.IS_DEBUG: try:
x = json.loads( return datetime.strptime(str(entry.get('eventDate')), GPV_DATE_FORMAT)
"""{ except ValueError:
"current": return datetime.max
{
"gav":
{
"message": "Черга спеціальних аварійних відключень (СГАВ):",
"queue": null
},
"hasQueue": "yes",
"note": "Станом на 21:05 14.01.2026 за вказаним особовим рахунком '30014180' споживач підпадає під чергу '5.1' Графіку погодинного відключення(ГПВ)",
"queue": 5,
"sgav":
{
"message": "Черга спеціальних аварійних відключень (СГАВ):",
"queue": null
},
"subQueue": 1
},
"schedule":
[
{
"createdAt": "14.01.2026 19:11",
"eventDate": "15.04.2026",
"queues":
{
"5.1":
[
{
"from": "07:30",
"shutdownHours": "07:30-11:00",
"status": 1,
"to": "11:00"
},
{
"from": "14:30",
"shutdownHours": "14:30-20:00",
"status": 1,
"to": "20:00"
},
{
"from": "22:00",
"shutdownHours": "22:00-00:00",
"status": 1,
"to": "00:00"
}
]
},
"scheduleApprovedSince": "14.04.2026 19:11"
},
{
"createdAt": "13.04.2026 20:28",
"eventDate": "14.04.2026",
"queues":
{
"5.1":
[
{
"from": "03:00",
"shutdownHours": "03:00-07:00",
"status": 1,
"to": "07:00"
},
{
"from": "11:00",
"shutdownHours": "11:00-17:30",
"status": 1,
"to": "17:30"
}
]
},
"scheduleApprovedSince": "14.04.2026 09:56"
}
]
}
"""
)
else:
response = get_GPV_response_from_oe(cfg.ACCOUNT)
if response.status_code != 200:
print("code=" + str(response.status_code))
return STATUS_ERROR_GPV_RESPONSE
x = response.json()
outage_queue = str(x["current"]["queue"]) def parse_gpv_data(x) -> List[GPVData]:
if x["current"]["subQueue"]: if not isinstance(x, dict):
outage_queue += '.' + str(x["current"]["subQueue"]) raise ValueError(f'Unexpected GPV response type: {type(x).__name__}')
datas = '' current = x.get('current') or {}
if 'queues' in str(x["schedule"]): if current.get('queue') is None:
datas = sorted(x["schedule"], key=lambda item: item['eventDate']) LOG.warning(f'No outage queue in GPV response: {current}')
return []
outage_queue = str(current['queue'])
if current.get('subQueue'):
outage_queue += '.' + str(current['subQueue'])
for entry in datas: schedule = x.get('schedule')
hours_off = [] if not isinstance(schedule, list):
hours_on = [] return []
if len(entry) > 0:
approved_from = 'Запроваджено ' + entry['scheduleApprovedSince']
event_date = entry['eventDate']
hours_list = entry['queues'][outage_queue]
else:
hours_list = []
approved_from = 'Hемає даних про дату запровадження'
event_date = datetime.today().strftime('%Y-%m-%d')
res: List[GPVData] = []
for entry in sorted((e for e in schedule if isinstance(e, dict)), key=gpv_date_key):
queues = entry.get('queues')
event_date = entry.get('eventDate')
if not isinstance(queues, dict) or not event_date:
continue
if outage_queue not in queues:
LOG.warning(f'No queue {outage_queue} in GPV schedule for {event_date}')
continue
approved_since = entry.get('scheduleApprovedSince')
approved_from = f'Запроваджено {approved_since}' if approved_since else 'Hемає даних про дату запровадження'
hours_list = queues.get(outage_queue) or []
if cfg.IS_DEBUG: if cfg.IS_DEBUG:
print(f'hours_list{hours_list}, approved_from={approved_from}, event_date={event_date}') print(f'hours_list{hours_list}, approved_from={approved_from}, event_date={event_date}')
hours_off = []
hours_on = []
for h in hours_list: for h in hours_list:
if h["status"] == 1: if isinstance(h, dict) and h.get("status") == 1 and h.get("from") and h.get("to"):
hours_off.append(h["from"]) hours_off.append(h["from"])
hours_on.append(h["to"]) hours_on.append(h["to"])
@@ -346,145 +321,177 @@ async def process_gpv():
print(f'hours_off = {hours_off}') print(f'hours_off = {hours_off}')
print(f'hours_on = {hours_on}') print(f'hours_on = {hours_on}')
update_gpv_data(PARSED_GPV_DATAS, GPVData(approved_from=approved_from, event_date=event_date, hours_off=hours_off, hours_on=hours_on, outage_queue=outage_queue)) gpv_data = GPVData(approved_from=approved_from, event_date=event_date, hours_off=hours_off,
hours_on=hours_on, outage_queue=outage_queue)
if gpv_data.actual_till_ts is None:
LOG.warning(f'GPV entry skipped due to wrong event date: {gpv_data}')
continue
res.append(gpv_data)
return res
check_and_send_gpv_message()
def fetch_gpv_json():
if cfg.IS_DEBUG:
with open(os.path.join(cfg.TEST_DIR, 'data_gpv.json'), 'r', encoding='utf-8') as f:
return json.load(f)
if not cfg.ACCOUNT:
LOG.error('Account number is not configured')
return None
response = get_GPV_response_from_oe(cfg.ACCOUNT)
if response.status_code != 200:
LOG.error(f'GPV response code={response.status_code}')
return None
return response.json()
async def process_gpv():
try:
x = await asyncio.to_thread(fetch_gpv_json)
if x is None:
return STATUS_ERROR_GPV_RESPONSE
new_data = parse_gpv_data(x)
except (requests.RequestException, ValueError, OSError) as e:
LOG.error(f'Error getting GPV data: {safe_error(e)}')
return STATUS_ERROR_GPV_RESPONSE
for el in new_data:
update_gpv_data(PARSED_GPV_DATAS, el)
remove_old_gpv_data(PARSED_GPV_DATAS)
await check_and_send_gpv_message()
return STATUS_SEND_OK return STATUS_SEND_OK
def check_and_send_gpv_message(): async def check_and_send_gpv_message():
for el in PARSED_GPV_DATAS: now_ts = datetime.now().timestamp()
for el in list(PARSED_GPV_DATAS):
if el.need_to_send and (el.actual_till_ts + 3600.0 >= now_ts):
message = get_GPV_message(el.approved_from, el.event_date, el.hours_off, el.hours_on, el.outage_queue) message = get_GPV_message(el.approved_from, el.event_date, el.hours_off, el.hours_on, el.outage_queue)
if el.need_to_send and (el.start_ts[-1] + 3600.0 >= datetime.now().timestamp()): if await asyncio.to_thread(notify, message):
tg_response = send_message_to_tg_get(message)
if tg_response == 200:
el.set_send() el.set_send()
LOG.debug(f'TG message send {message}')
else:
LOG.error(f'TG send error code: {STATUS_ERROR_SEND_TO_TG}')
if cfg.USE_MATRIX and not cfg.IS_DEBUG:
res = send_matrix_notification(message)
if not res:
LOG.error(f'MX send error code: {STATUS_ERROR_SEND_TO_MX}')
def update_gpv_data(data_list: List[GPVData], new_data: GPVData): def update_gpv_data(data_list: List[GPVData], new_data: GPVData):
# find by TS # find by event date and queue
obj = next((obj for obj in data_list if obj.start_ts == new_data.start_ts), None) old_data = next((obj for obj in data_list if obj.key == new_data.key), None)
if obj: if old_data is None:
# if exists - update: data_list.append(new_data)
old_data = data_list[data_list.index(obj)] LOG.debug(f'New element appended to list {new_data}')
if not old_data.need_to_send: # notification already send elif old_data.need_to_send:
if old_data.start_ts != new_data.start_ts: # notification is not sent yet - just take the latest data
data_list[data_list.index(obj)] = new_data data_list[data_list.index(old_data)] = new_data
elif not old_data.is_same_schedule(new_data):
# schedule is changed after notification - it will be sent again
data_list[data_list.index(old_data)] = new_data
LOG.debug(f'Updated element {old_data} in list with new data {new_data}') LOG.debug(f'Updated element {old_data} in list with new data {new_data}')
else: else:
LOG.debug(f'Element {old_data} was not updated due to the same') LOG.debug(f'Element {old_data} was not updated due to the same')
else:
data_list[data_list.index(obj)] = new_data
else:
# if not exists - add
data_list.append(new_data)
LOG.debug(f'New element appended to list {new_data}')
# data_list SORT
# remove OLD entries
def remove_old_gpv_data(data_list: List[GPVData]):
ts = datetime.now().timestamp() ts = datetime.now().timestamp()
for el in data_list[:]: for el in data_list[:]:
if el.start_ts and (el.start_ts[-1] + 1800) < ts: if el.actual_till_ts is None or (el.actual_till_ts + 1800) < ts:
LOG.debug(f'Old element removed from list {el}') LOG.debug(f'Old element removed from list {el}')
data_list.remove(el) data_list.remove(el)
def get_message_with_po(el: POData): def get_message_with_po(el: POData):
address = get_address(el.city, el.street, el.house) address = html.escape(get_address(el.city, el.street, el.house))
if el.need_to_send: if el.need_to_send:
message = ( message = (
f'\U000026A0 <b>Увага!</b>\nНа {el.start_date} за адресою {address} заплановано <b>{el.po_type}</b> відключення (заявка від {el.placement_date}).\n' f'\U000026A0 <b>Увага!</b>\nНа {html.escape(el.start_date)} за адресою {address} заплановано '
f'Причина відключення: {el.reason}\n' f'<b>{html.escape(el.po_type)}</b> відключення (заявка від {html.escape(el.placement_date)}).\n'
f'Початок вимкнення <b>{el.start_date} об {el.start_time}</b>\n' f'Причина відключення: {html.escape(el.reason)}\n'
f'Кінець вимкнення <b>{el.stop_date} об {el.stop_time}</b>') f'Початок вимкнення <b>{html.escape(el.start_date)} об {html.escape(el.start_time)}</b>\n'
f'Кінець вимкнення <b>{html.escape(el.stop_date)} об {html.escape(el.stop_time)}</b>')
else: else:
message = f'На {el.start_date} Планових вимкнень немає \U0001F44C \U0001F483' message = f'На {html.escape(el.start_date)} Планових вимкнень немає \U0001F44C \U0001F483'
return message return message
async def process_po(): def fetch_po_root():
if not cfg.IS_DEBUG: if cfg.IS_DEBUG:
with open(os.path.join(cfg.TEST_DIR, 'data_po_2days_02.html'), 'r', encoding='utf-8') as f:
page = f.read()
else:
response = get_PO_response() response = get_PO_response()
if response.status_code != 200: if response.status_code != 200:
LOG.error(f'PO response code={response.status_code}')
return None
page = response.text
if not page.strip():
LOG.error('Empty PO page')
return None
return etree.fromstring(page, etree.HTMLParser())
async def process_po():
try:
root = await asyncio.to_thread(fetch_po_root)
if root is None:
return STATUS_ERROR_PO_RESPONSE return STATUS_ERROR_PO_RESPONSE
html = response.text
parser = etree.HTMLParser()
root = etree.parse(StringIO(html), parser)
else:
# html = "<html><body><div id='shutdowns-table'><div class='table'><div class='flex table-header'><div class='region'>Район</div><div class='city'>Населений пункт</div><div class='street'>Вулиця</div><div class='house_number'>№ буд</div>" \
# "<div class='building_part_number'>Корпус</div><div class='dispatch_name'>Диспетчерська назва об'єкту</div><div class='type-shutdown'>Вид робіт</div><div class='reason-shutdown'>Причина робіт</div><div class='placement_date'>Дата розміщення</div>" \
# "<div class='shutdown_date'>Дата і час вимкнення</div><div class='turn_on_date'>Дата і час включення</div><div class='next_date'>Наступна дата робіт</div></div></div><div class='shutdowns'><div><div><div class='table-row flex'><div class='region'>Піднянський район</div>" \
# "<div class='city'>Гірна</div><div class='street'>Тевського</div><div class='house_number'>9</div><div class='building_part_number'/><div class='dispatch_name'>ДСП 110/Пр.&quot;ЗБВIК&quot;/30-413/Л-6</div><div class='type-shutdown'>Планове</div>" \
# "<div class='reason-shutdown reason-text-container'><div class='reason-text-hide'>Поточний ремонт</div></div><div class='placement_date'><div class='date'>2025.03.24</div><div class='time'>11:50</div></div><div class='shutdown_date'><div class='date'>2026.04.10</div>" \
# "<div class='time'>09:00</div></div><div class='turn_on_date'><div class='date'>2026.04.10</div><div class='time'>18:00</div></div><div class='next_date'><div class='reason-text-hide'/></div></div></div></div></div></div></body></html>"
with open('test/data_po_2days_02.html', 'r') as file:
html = file.read()
root = etree.fromstring(html)
global PARSED_PO_DATAS
new_data = parse_po_data(root) new_data = parse_po_data(root)
for el in new_data: except (requests.RequestException, etree.LxmlError, ValueError, OSError) as e:
await update_po_data(PARSED_PO_DATAS, el) LOG.error(f'Error getting PO data: {safe_error(e)}')
return STATUS_ERROR_PO_RESPONSE
check_and_send_po_message() for el in new_data:
update_po_data(PARSED_PO_DATAS, el)
remove_old_po_data(PARSED_PO_DATAS)
await check_and_send_po_message()
return STATUS_SEND_OK return STATUS_SEND_OK
def check_and_send_po_message(): async def check_and_send_po_message():
for el in PARSED_PO_DATAS: for el in list(PARSED_PO_DATAS):
if el.need_to_send: if el.need_to_send:
message = get_message_with_po(el) message = get_message_with_po(el)
tg_response = send_message_to_tg_get(message) if await asyncio.to_thread(notify, message):
if tg_response == 200:
el.set_sent() el.set_sent()
el.set_notified_now() el.set_notified_now()
LOG.debug(f'TG message send {message}')
else:
LOG.error(f'TG send error code: {STATUS_ERROR_SEND_TO_TG}')
if cfg.USE_MATRIX:
res = send_matrix_notification(message)
if not res:
LOG.error(f'MX send error code: {STATUS_ERROR_SEND_TO_MX}')
async def update_po_data(data_list:List[POData], new_data: POData): def get_daily_reminder_time(now: datetime) -> datetime:
# find by TS """Today's moment of the first data processing - notifications are repeated after it"""
obj = next((obj for obj in data_list if obj.start_ts == new_data.start_ts), None) first_minute = min(cfg.PROCESSING_MINUTES) if cfg.PROCESSING_MINUTES else 0
if obj: return now.replace(hour=cfg.PROCESS_START_HOUR, minute=first_minute, second=0, microsecond=0)
# if exists - update
old_data = data_list[data_list.index(obj)]
if not old_data.need_to_send: # notification already send def is_reminder_needed(old_data: POData, now: Optional[datetime] = None) -> bool:
if old_data != new_data: now = now or datetime.now()
# update 'need_to_send' to re-send notification every day if old_data.notified_at is None:
if ((not old_data.notified_at + timedelta(days=1) <= datetime.now()) or return True
# or if current hour - is the start data processing hour with firs minute in cfg.PROCESSING_MINUTES list # re-send notification every day
(cfg.PROCESS_START_HOUR and datetime.now().strftime('%H') == cfg.PROCESS_START_HOUR and datetime.now().strftime('%M') == cfg.PROCESSING_MINUTES[0])): if old_data.notified_at + timedelta(days=1) <= now:
new_data.set_sent() return True
new_data.update_notified_at(old_data) # or if the first data processing of the day is passed after the last notification
data_list[data_list.index(obj)] = new_data reminder_time = get_daily_reminder_time(now)
return old_data.notified_at < reminder_time <= now
def update_po_data(data_list: List[POData], new_data: POData):
# find by start TS and address
old_data = next((obj for obj in data_list if obj.key == new_data.key), None)
if old_data is None:
data_list.append(new_data)
LOG.debug(f'New element appended to list {new_data}')
elif old_data.need_to_send:
# notification is not sent yet - just take the latest data
data_list[data_list.index(old_data)] = new_data
elif old_data != new_data or is_reminder_needed(old_data):
# data changed or it is time to remind - notification will be sent again
data_list[data_list.index(old_data)] = new_data
LOG.debug(f'Updated element {old_data} in list with new data {new_data}') LOG.debug(f'Updated element {old_data} in list with new data {new_data}')
else: else:
LOG.debug(f'Element {old_data} was not updated due to the same') LOG.debug(f'Element {old_data} was not updated due to the same')
else:
data_list[data_list.index(obj)] = new_data
else:
# if not exists - add
data_list.append(new_data)
LOG.debug(f'New element appended to list {new_data}')
# data_list SORT
# remove OLD entries
def remove_old_po_data(data_list: List[POData]):
ts = datetime.now().timestamp() ts = datetime.now().timestamp()
for el in data_list[:]: for el in data_list[:]:
if el.start_ts and (el.start_ts + 3600) <= ts: if el.start_ts and (el.start_ts + 3600) <= ts: