feat: integrate Google Scholar alerts and improve paper filtering pipeline
- Add Google Scholar Alert collection via Gmail IMAP - Add Gmail App Password based authentication - Add Google Scholar environment variables to NAS compose configuration - Enable Google Scholar Alert source in config.nas.yaml - Disable subject filtering that incorrectly excluded Scholar alert emails - Parse paper titles and links from Google Scholar alert HTML emails - Deduplicate collected Scholar papers by normalized title - Filter Scholar UI/control links such as update alert and unsubscribe entries - Filter bracket-only alert labels such as [automotive radar] - Add Google Scholar specific relevance threshold - Keep global min_relevance at 2 - Set Google Scholar minimum relevance to 3 - Reduce Scholar candidates from 229 collected / 207 merged to 28 accepted - Add Gemini API rate limiting - Enforce minimum 13 second interval between Gemini requests - Apply shared rate limiter to abstract and full-text enrichment - Prevent Gemini free-tier 5 RPM quota errors - Verify retry processing of previously failed AI enrichment jobs - Verify end-to-end NAS workflow - Google Scholar Alert collection successful - Gemini enrichment successful without HTTP 429 errors - Joplin report generation and WebDAV synchronization successful
This commit is contained in:
+6
-5
@@ -3,6 +3,7 @@ app:
|
||||
lookback_days: 14
|
||||
max_papers_per_source_per_query: 20
|
||||
min_relevance: 2
|
||||
google_scholar_min_relevance: 3
|
||||
output_dir: ./data/outbox
|
||||
database_path: ./data/papers.db
|
||||
log_level: INFO
|
||||
@@ -55,18 +56,18 @@ sources:
|
||||
base_url: https://api.semanticscholar.org/graph/v1/paper/search
|
||||
|
||||
crossref:
|
||||
enabled: true
|
||||
enabled: true
|
||||
mailto_env: CROSSREF_MAILTO
|
||||
base_url: https://api.crossref.org/works
|
||||
|
||||
google_scholar_alert:
|
||||
enabled: false
|
||||
gmail_address_env: GMAIL_ADDRESS
|
||||
gmail_app_password_env: GMAIL_APP_PASSWORD
|
||||
enabled: true
|
||||
gmail_address_env: GOOGLE_SCHOLAR_GMAIL_ADDRESS
|
||||
gmail_app_password_env: GOOGLE_SCHOLAR_GMAIL_APP_PASSWORD
|
||||
imap_host: imap.gmail.com
|
||||
mailbox: INBOX
|
||||
sender_contains: scholaralerts-noreply
|
||||
subject_contains: Google Scholar
|
||||
subject_contains: ""
|
||||
|
||||
ai:
|
||||
enabled: true
|
||||
|
||||
Reference in New Issue
Block a user