feat: integrate Google Scholar alerts and improve paper filtering pipeline

- Add Google Scholar Alert collection via Gmail IMAP
- Add Gmail App Password based authentication
- Add Google Scholar environment variables to NAS compose configuration
- Enable Google Scholar Alert source in config.nas.yaml
- Disable subject filtering that incorrectly excluded Scholar alert emails
- Parse paper titles and links from Google Scholar alert HTML emails
- Deduplicate collected Scholar papers by normalized title
- Filter Scholar UI/control links such as update alert and unsubscribe entries
- Filter bracket-only alert labels such as [automotive radar]

- Add Google Scholar specific relevance threshold
- Keep global min_relevance at 2
- Set Google Scholar minimum relevance to 3
- Reduce Scholar candidates from 229 collected / 207 merged to 28 accepted

- Add Gemini API rate limiting
- Enforce minimum 13 second interval between Gemini requests
- Apply shared rate limiter to abstract and full-text enrichment
- Prevent Gemini free-tier 5 RPM quota errors
- Verify retry processing of previously failed AI enrichment jobs

- Verify end-to-end NAS workflow
- Google Scholar Alert collection successful
- Gemini enrichment successful without HTTP 429 errors
- Joplin report generation and WebDAV synchronization successful
This commit is contained in:
2026-08-21 00:25:24 +09:00
parent c3aad03182
commit 2543b0c23f
8 changed files with 233 additions and 51 deletions
+20
View File
@@ -1,5 +1,21 @@
import json
import time
_GEMINI_MIN_INTERVAL = 13.0
_GEMINI_LAST_CALL = 0.0
def _wait_for_gemini_rate_limit():
global _GEMINI_LAST_CALL
now = time.monotonic()
elapsed = now - _GEMINI_LAST_CALL
wait = _GEMINI_MIN_INTERVAL - elapsed
if wait > 0:
time.sleep(wait)
_GEMINI_LAST_CALL = time.monotonic()
SYSTEM = '''You are a technical literature triage assistant for an automotive radar engineer.
@@ -111,6 +127,8 @@ def enrich_with_gemini(paper, api_key, model):
+ json.dumps(payload, ensure_ascii=False)
)
_wait_for_gemini_rate_limit()
response = client.models.generate_content(
model=model,
contents=prompt,
@@ -172,6 +190,8 @@ contribution, and relevance to automotive radar when supported by the text.
+ json.dumps(payload, ensure_ascii=False)
)
_wait_for_gemini_rate_limit()
response = client.models.generate_content(
model=model,
contents=prompt,