feat: integrate Google Scholar alerts and improve paper filtering pipeline
- Add Google Scholar Alert collection via Gmail IMAP - Add Gmail App Password based authentication - Add Google Scholar environment variables to NAS compose configuration - Enable Google Scholar Alert source in config.nas.yaml - Disable subject filtering that incorrectly excluded Scholar alert emails - Parse paper titles and links from Google Scholar alert HTML emails - Deduplicate collected Scholar papers by normalized title - Filter Scholar UI/control links such as update alert and unsubscribe entries - Filter bracket-only alert labels such as [automotive radar] - Add Google Scholar specific relevance threshold - Keep global min_relevance at 2 - Set Google Scholar minimum relevance to 3 - Reduce Scholar candidates from 229 collected / 207 merged to 28 accepted - Add Gemini API rate limiting - Enforce minimum 13 second interval between Gemini requests - Apply shared rate limiter to abstract and full-text enrichment - Prevent Gemini free-tier 5 RPM quota errors - Verify retry processing of previously failed AI enrichment jobs - Verify end-to-end NAS workflow - Google Scholar Alert collection successful - Gemini enrichment successful without HTTP 429 errors - Joplin report generation and WebDAV synchronization successful
This commit is contained in:
@@ -1,5 +1,21 @@
|
||||
import json
|
||||
import time
|
||||
|
||||
_GEMINI_MIN_INTERVAL = 13.0
|
||||
_GEMINI_LAST_CALL = 0.0
|
||||
|
||||
|
||||
def _wait_for_gemini_rate_limit():
|
||||
global _GEMINI_LAST_CALL
|
||||
|
||||
now = time.monotonic()
|
||||
elapsed = now - _GEMINI_LAST_CALL
|
||||
wait = _GEMINI_MIN_INTERVAL - elapsed
|
||||
|
||||
if wait > 0:
|
||||
time.sleep(wait)
|
||||
|
||||
_GEMINI_LAST_CALL = time.monotonic()
|
||||
|
||||
SYSTEM = '''You are a technical literature triage assistant for an automotive radar engineer.
|
||||
|
||||
@@ -111,6 +127,8 @@ def enrich_with_gemini(paper, api_key, model):
|
||||
+ json.dumps(payload, ensure_ascii=False)
|
||||
)
|
||||
|
||||
_wait_for_gemini_rate_limit()
|
||||
|
||||
response = client.models.generate_content(
|
||||
model=model,
|
||||
contents=prompt,
|
||||
@@ -172,6 +190,8 @@ contribution, and relevance to automotive radar when supported by the text.
|
||||
+ json.dumps(payload, ensure_ascii=False)
|
||||
)
|
||||
|
||||
_wait_for_gemini_rate_limit()
|
||||
|
||||
response = client.models.generate_content(
|
||||
model=model,
|
||||
contents=prompt,
|
||||
|
||||
Reference in New Issue
Block a user