feat: integrate Google Scholar alerts and improve paper filtering pipeline

- Add Google Scholar Alert collection via Gmail IMAP
- Add Gmail App Password based authentication
- Add Google Scholar environment variables to NAS compose configuration
- Enable Google Scholar Alert source in config.nas.yaml
- Disable subject filtering that incorrectly excluded Scholar alert emails
- Parse paper titles and links from Google Scholar alert HTML emails
- Deduplicate collected Scholar papers by normalized title
- Filter Scholar UI/control links such as update alert and unsubscribe entries
- Filter bracket-only alert labels such as [automotive radar]

- Add Google Scholar specific relevance threshold
- Keep global min_relevance at 2
- Set Google Scholar minimum relevance to 3
- Reduce Scholar candidates from 229 collected / 207 merged to 28 accepted

- Add Gemini API rate limiting
- Enforce minimum 13 second interval between Gemini requests
- Apply shared rate limiter to abstract and full-text enrichment
- Prevent Gemini free-tier 5 RPM quota errors
- Verify retry processing of previously failed AI enrichment jobs

- Verify end-to-end NAS workflow
- Google Scholar Alert collection successful
- Gemini enrichment successful without HTTP 429 errors
- Joplin report generation and WebDAV synchronization successful
This commit is contained in:
2026-08-21 00:25:24 +09:00
parent c3aad03182
commit 2543b0c23f
8 changed files with 233 additions and 51 deletions
+6 -5
View File
@@ -3,6 +3,7 @@ app:
lookback_days: 14
max_papers_per_source_per_query: 20
min_relevance: 2
google_scholar_min_relevance: 3
output_dir: ./data/outbox
database_path: ./data/papers.db
log_level: INFO
@@ -55,18 +56,18 @@ sources:
base_url: https://api.semanticscholar.org/graph/v1/paper/search
crossref:
enabled: true
enabled: true
mailto_env: CROSSREF_MAILTO
base_url: https://api.crossref.org/works
google_scholar_alert:
enabled: false
gmail_address_env: GMAIL_ADDRESS
gmail_app_password_env: GMAIL_APP_PASSWORD
enabled: true
gmail_address_env: GOOGLE_SCHOLAR_GMAIL_ADDRESS
gmail_app_password_env: GOOGLE_SCHOLAR_GMAIL_APP_PASSWORD
imap_host: imap.gmail.com
mailbox: INBOX
sender_contains: scholaralerts-noreply
subject_contains: Google Scholar
subject_contains: ""
ai:
enabled: true